<?xml version="1.0" encoding="utf-8"?>
<feed xml:lang="en" xmlns="http://www.w3.org/2005/Atom"><title>Recent posts to news</title><link href="https://sourceforge.net/p/shiftc/news/" rel="alternate"/><link href="https://sourceforge.net/p/shiftc/news/feed.atom" rel="self"/><id>https://sourceforge.net/p/shiftc/news/</id><updated>2024-03-15T17:26:33.247000Z</updated><subtitle>Recent posts to news</subtitle><entry><title>Shift 8.2 released</title><link href="https://sourceforge.net/p/shiftc/news/2024/03/shift-82-released/" rel="alternate"/><published>2024-03-15T17:26:33.247000Z</published><updated>2024-03-15T17:26:33.247000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.net543b14b37fa9f9ac1d05966bcec0ecb22865989a</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Fixed collection and application of default acls in shift-bin&lt;br/&gt;
    - Added additional documentation in shiftc&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 8.1 released</title><link href="https://sourceforge.net/p/shiftc/news/2024/01/shift-81-released/" rel="alternate"/><published>2024-01-09T01:43:28.127000Z</published><updated>2024-01-09T01:43:28.127000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.net3450e46451a10f5333509bfbb4aca57158817e9d</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Added silent corruption database search using --search with --last-sum&lt;br/&gt;
    - Added custom lustre striping of directories using DR in expressions&lt;br/&gt;
    - Added alternative metadata locking for broken NFSv3 locks&lt;br/&gt;
    - Added debugging output when manager unable to map local/remote files&lt;br/&gt;
    - Changed embedded mesh handling to latest mesh version&lt;br/&gt;
    - Fixed fish-tcp --secure SSL handling with more recent IO::Socket::SSL&lt;br/&gt;
    - Fixed mount collection when file server is an alias to many IPs&lt;br/&gt;
    - Fixed handling of &lt;a class="user-mention" href="/u/kfi/"&gt;@kfi&lt;/a&gt; forms in lustre mount collection&lt;br/&gt;
    - Fixed remote mount point in GPFS mount collection&lt;br/&gt;
    - Fixed mcp default stripe handling overriding --stripe=0&lt;br/&gt;
    - Fixed handling of --mgr so no longer always overrides with default&lt;br/&gt;
    - Fixed inherited retry count of operations in run state during restart&lt;br/&gt;
    - Fixed clean of files associated with old monitoring processes&lt;br/&gt;
    - Fixed exception during silent corruption processing of some tar operations&lt;br/&gt;
    - Fixed escape of commas in email command line&lt;br/&gt;
    - Fixed synchronization of monitor files between managers&lt;br/&gt;
    - Fixed error handling of manager invocations&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 8.0 released</title><link href="https://sourceforge.net/p/shiftc/news/2021/09/shift-80-released/" rel="alternate"/><published>2021-09-17T00:33:31.869000Z</published><updated>2021-09-17T00:33:31.869000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.netc1e8dfa58dd05b4426324da5e88b190a56be1a35</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Note that metadata is not backward compatible with previous versions&lt;br/&gt;
    - Added plots by tool, subnet, and batch size&lt;br/&gt;
    - Added detection of BeeGFS file systems&lt;br/&gt;
    - Changed plots from line to heatmap for improved scalability&lt;br/&gt;
    - Changed processing so batches only contain single operation type&lt;br/&gt;
    - Changed monitoring to support manager hosts across shared file system&lt;br/&gt;
    - Changed naming of built-in tools in --stats and new --plot by tool&lt;br/&gt;
    - Fixed detection of DMF file systems when using xdsm mount option&lt;br/&gt;
    - Fixed detection of GPFS file systems when mmlsmgr not user-accessible&lt;br/&gt;
    - Fixed --wait status retrieval when integrated with Mesh framework&lt;br/&gt;
    - Fixed shift-bin striping with lustre progressive layouts when --stripe=0&lt;br/&gt;
    - Fixed use of XS Data::MessagePack when embedded pure perl version differs&lt;br/&gt;
    - Fixed scalability of built-in yEnc encoding/decoding&lt;br/&gt;
    - Fixed unnecessary metadata traversal when showing detailed status&lt;br/&gt;
    - Fixed application of lustre striping expression to directories&lt;br/&gt;
    - Removed bbcp and gridftp support since reported fatal bugs never fixed&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 7.1 released</title><link href="https://sourceforge.net/p/shiftc/news/2021/09/shift-71-released/" rel="alternate"/><published>2021-09-17T00:32:46.869000Z</published><updated>2021-09-17T00:32:46.869000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.neta9d571b476e784aa435629d19d8effacb6bfd0f1</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Note that the shiftc included with 7.0 was the wrong version (still 6.3)&lt;br/&gt;
    - Added latency collection on windows systems&lt;br/&gt;
    - Added tcp buffer size collection on osx&lt;br/&gt;
    - Added periodic latency measurement to track network issues&lt;br/&gt;
    - Added support of \Q metacharacter in --include/--exclude&lt;br/&gt;
    - Added manager setting for maximum files per batch&lt;br/&gt;
    - Added manager setting for time between latency measurement&lt;br/&gt;
    - Added origin IP address to transfer metadata&lt;br/&gt;
    - Added save of manager exceptions to associated user debug file&lt;br/&gt;
    - Added --plot by file system&lt;br/&gt;
    - Changed tracking of built-in operations for stats output&lt;br/&gt;
    - Changed file metadata operations to report metadata rate&lt;br/&gt;
    - Changed corruption tracking to not count file sizes differing&lt;br/&gt;
    - Fixed globs while locating ssh agents to reduce system audit messages&lt;br/&gt;
    - Fixed potential for multiple simultaneous keepalive invocations&lt;br/&gt;
    - Fixed ephemeral port usage during remote bbftpd invocation&lt;br/&gt;
    - Fixed missing handling of --no-recall in some scenarios&lt;br/&gt;
    - Fixed accuracy of rsync rates during partial file transfers&lt;br/&gt;
    - Fixed failure to detect completion properly in some tar scenarios&lt;br/&gt;
    - Fixed parallelization due to overwrite of per user mount metadata&lt;br/&gt;
    - Fixed manager exception during read of corrupt/incomplete gzi files&lt;br/&gt;
    - Fixed handling of manager disk exhaustion to prevent finalized metadata&lt;br/&gt;
    - Fixed potential divide by zero exceptions during throttling&lt;br/&gt;
    - Fixed lustre handling in shift-bin to current lustre API&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 7.0 released</title><link href="https://sourceforge.net/p/shiftc/news/2019/10/shift-70-released/" rel="alternate"/><published>2019-10-04T21:51:30.670000Z</published><updated>2019-10-04T21:51:30.670000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.netc32bbc2e1aaf2c4fcf592b761861a82e788e2e32</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Note that metadata is not backward compatible with previous versions&lt;br/&gt;
    - Fixed vulnerability in root --stats/--status due to use of Storable when metadata directly accessible by users (bug report by J. Neff)&lt;br/&gt;
    - Fixed race condition causing permission denied during some remote mkdirs&lt;br/&gt;
    - Fixed built-in checksums when thread creation fails&lt;br/&gt;
    - Fixed chattrs when thread creation fails&lt;br/&gt;
    - Fixed leftover shift-bin processes due to unneeded chattr thread spawns&lt;br/&gt;
    - Fixed exception when setting binmode on remote file handles&lt;br/&gt;
    - Fixed local dmget calls during traversal when source is not local&lt;br/&gt;
    - Fixed duplicated source tar lines in dmget input files&lt;br/&gt;
    - Fixed deceptive thread creation errors during fish-tcp initialization&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 6.3 released</title><link href="https://sourceforge.net/p/shiftc/news/2019/09/shift-63-released/" rel="alternate"/><published>2019-09-18T22:19:43.915000Z</published><updated>2019-09-18T22:19:43.915000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.netaf20a125f885c257f30b69095bfe097702870c37</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Fixed erroneous crontab entries for extra clients when --clients &amp;gt; 1&lt;br/&gt;
    - Fixed slash prepended to relative paths during tar creation&lt;br/&gt;
    - Fixed Tie::DB_FileLock exceptions that can abort manager updates&lt;br/&gt;
    - Fixed tainted file names preventing manager synchronization&lt;br/&gt;
    - Fixed missing lock during writes of user mount db&lt;br/&gt;
    - Fixed use of dequeue_timed preventing directory traversal in older perls&lt;br/&gt;
    - Fixed double locks causing deadlock during manager synchronization&lt;br/&gt;
    - Fixed thread race condition resulting in zombie shift-aux chattr processes&lt;br/&gt;
    - Fixed rounding of lustre stripe count causing allocated stripes off by 1&lt;br/&gt;
    - Fixed potential noop in shift-aux chattr when thread creation fails&lt;br/&gt;
    - Fixed stalls due to deletion of local mount information during restarts&lt;br/&gt;
    - Fixed asymmetric fish-tcp performance during some wan scenarios&lt;br/&gt;
    - Fixed potential duplicate error messages in various transports&lt;br/&gt;
    - Fixed duplication of corrupt byte ranges resulting in negative rates/sizes&lt;br/&gt;
    - Fixed infinite retry of corruption rectification when no progress made&lt;br/&gt;
    - Fixed missing write error handling in fish and fish-tcp transports&lt;br/&gt;
    - Fixed metadata counts during tar creation renames with --restart=ignore&lt;br/&gt;
    - Fixed bad newline handling in shift-aux escape/unescape commands&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 6.2 released</title><link href="https://sourceforge.net/p/shiftc/news/2019/06/shift-62-released/" rel="alternate"/><published>2019-06-25T00:05:43.141000Z</published><updated>2019-06-25T00:05:43.141000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.net0a5037185c202b9f951971d8127e5a63f22e5bd8</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Added multi-threading of directory traversal&lt;br/&gt;
    - Added multi-threading of attr preservation when external programs invoked&lt;br/&gt;
    - Added --interval to dynamically adjust batch size around fixed frequency&lt;br/&gt;
    - Added use of gzip index files to increase metadata read performance&lt;br/&gt;
    - Added --no-silent to disable silent corruption detection&lt;br/&gt;
    - Added ability to escape multiple lines over stdin in shift-aux&lt;br/&gt;
    - Added syntax checking of numeric options that can take units&lt;br/&gt;
    - Changed --files and --size to be minimums instead of absolutes&lt;br/&gt;
    - Changed time remaining to more accurately reflect manager overhead&lt;br/&gt;
    - Fixed remote host selection that was running per file instead of per batch&lt;br/&gt;
    - Fixed detection of operations already performed by another client&lt;br/&gt;
    - Fixed slow processing of doing logs due to small backward read size&lt;br/&gt;
    - Fixed local/remote fadvise that had old initial implementation&lt;br/&gt;
    - Fixed --no-recall handling of remote files&lt;br/&gt;
    - Fixed possible exceptions during some thread joins&lt;br/&gt;
    - Fixed Net::Ping exceptions when Time::HiRes not available&lt;br/&gt;
    - Fixed exception when stat of remote file fails&lt;br/&gt;
    - Fixed use of setfacl when shift-bin available&lt;br/&gt;
    - Fixed run state when --restart=ignore ignores final operations&lt;br/&gt;
    - Fixed manager --doing=n so it retrieves nth doing in past&lt;br/&gt;
    - Fixed output of error table in stats due to tainted file names&lt;br/&gt;
    - Fixed option abbreviation so deprecated options don't invoke other options&lt;br/&gt;
    - Fixed double lustre striping during attribute preservation stage&lt;br/&gt;
    - Fixed high memory utilization during traversal of very large directories&lt;br/&gt;
    - Fixed stalls due to inadvertent newlines left in logs after restarts&lt;br/&gt;
    - Fixed preservation of lustre striping due to bad conditional&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 6.1 released</title><link href="https://sourceforge.net/p/shiftc/news/2019/02/shift-61-released/" rel="alternate"/><published>2019-02-08T02:01:31.188000Z</published><updated>2019-02-08T02:01:31.188000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.nete214f4785f2dc4fbff13f2fef922bc0c9445ac54</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Added manager config to globally disable use of cron for restarts&lt;br/&gt;
    - Fixed corruption tracking causing negative sizes/rates in --status&lt;br/&gt;
    - Fixed inadvertent call of test shift-mgr during mgr synchronization&lt;br/&gt;
    - Removed warning message when --no-cron specified&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 6.0 released</title><link href="https://sourceforge.net/p/shiftc/news/2019/01/shift-60-released/" rel="alternate"/><published>2019-01-17T01:31:51.513000Z</published><updated>2019-01-17T01:31:51.513000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.netcef7d3d1e3db7b8e0326904346b61374a0533087</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Note that this version is not backward compatible with previous versions&lt;br/&gt;
    - Added depth-first file stage processing using --pipeline&lt;br/&gt;
    - Added CSV history output using --history=csv&lt;br/&gt;
    - Added disablement of email status for states given in --no-mail&lt;br/&gt;
    - Added disablement of preservation for attributes given in --no-preserve&lt;br/&gt;
    - Added continuous real-time display of status using --monitor&lt;br/&gt;
    - Added error handling for manager disk exhaustion&lt;br/&gt;
    - Added overwrite of read-only files using --force&lt;br/&gt;
    - Added ability to ignore unrecoverable errors using --restart=ignore&lt;br/&gt;
    - Added faster get/set of external file attributes using shift-bin&lt;br/&gt;
    - Added plotting by client using --plot=client&lt;br/&gt;
    - Added alert state for items that must be externally checked/fixed&lt;br/&gt;
    - Added encrypted data streams to fish-tcp transport using --secure&lt;br/&gt;
    - Added better support for scp-style "user@" syntax&lt;br/&gt;
    - Added silent corruption detection for previously completed transfers&lt;br/&gt;
    - Added --last-sum option to query silent corruption database&lt;br/&gt;
    - Added fadvise of source after transfer when shift-bin available&lt;br/&gt;
    - Added per file dynamic striping via perl-based expressions&lt;br/&gt;
    - Added manager config for hash algorithm and hash leaf size&lt;br/&gt;
    - Added advanced --older/--newer expressions using atime/ctime/mtime&lt;br/&gt;
    - Added --doing manager option to see past/present doing logs&lt;br/&gt;
    - Added abort during tar creation if first split file already exists&lt;br/&gt;
    - Added same file detection across remote hosts&lt;br/&gt;
    - Changed --no-check option to --no-sanity&lt;br/&gt;
    - Changed manager --meta output to handle deeper data structures&lt;br/&gt;
    - Changed manager db_file config setting to mount_db&lt;br/&gt;
    - Changed dmget handling to use -a during tar creation to keep files online&lt;br/&gt;
    - Changed manager meta file format from base64 to yEnc&lt;br/&gt;
    - Changed manager log structured files to seekable compressed format&lt;br/&gt;
    - Changed manager doing logs to log structured format&lt;br/&gt;
    - Changed HMAC authentication from SHA1 to SHA512 in fish-tcp&lt;br/&gt;
    - Changed manager sync mechanism to better support multiple failovers&lt;br/&gt;
    - Changed estimated completion to use running avgs instead of elapsed time&lt;br/&gt;
    - Fixed mount detection for DMF file systems reported as dmapi&lt;br/&gt;
    - Fixed mount detection for lustre file systems with multiple MDS's&lt;br/&gt;
    - Fixed overwrite of symlinks when exist at destination&lt;br/&gt;
    - Fixed deprecated File::Glob usage on newer perls&lt;br/&gt;
    - Fixed Net::Ping invocations during latency measurements on redhat/centos&lt;br/&gt;
    - Fixed detection of automounted file systems&lt;br/&gt;
    - Fixed hang on PBS-controlled hosts during file system detection&lt;br/&gt;
    - Fixed --sync on DMF to copy file instead of summing when time/size match&lt;br/&gt;
    - Fixed DMF recalls during file overwrite when using built-in transports&lt;br/&gt;
    - Fixed incorrect striping when --stripe=0 given and transport is mcp&lt;br/&gt;
    - Fixed warning output in some --status=color scenarios&lt;br/&gt;
    - Fixed exception in detailed status when very large file with small files&lt;br/&gt;
    - Fixed race condition between check and creation of user metadata dir&lt;br/&gt;
    - Fixed tracking of corruption stats during --sync&lt;br/&gt;
    - Fixed stalls after host/process failures due to bad metadata recovery&lt;br/&gt;
    - Fixed throttling deadlock due to file not counting own preallocation&lt;br/&gt;
    - Fixed error detection in some gridftp scenarios&lt;br/&gt;
    - Removed manager lustre_default_stripe config setting&lt;br/&gt;
    - Removed manager min_split config setting&lt;br/&gt;
    - Removed previously allowed inverse options --preserve and --verify&lt;/p&gt;&lt;/div&gt;</summary></entry><entry><title>Shift 5.2 released</title><link href="https://sourceforge.net/p/shiftc/news/2018/04/shift-52-released/" rel="alternate"/><published>2018-04-23T22:47:51.362000Z</published><updated>2018-04-23T22:47:51.362000Z</updated><author><name>Paul Kolano</name><uri>https://sourceforge.net/u/pkolano/</uri></author><id>https://sourceforge.net0cec7a336f3e50ef9bca7d0fcd935e7b9f6c8317</id><summary type="html">&lt;div class="markdown_content"&gt;&lt;p&gt;CHANGES&lt;br/&gt;
    - Added ability to show stats in CSV format using --stats=csv&lt;br/&gt;
    - Fixed lfs setstripe calls to use -S instead of -s (lustre &amp;gt;= 2.3.0)&lt;br/&gt;
    - Fixed descent into directories matching --exclude during initialization&lt;br/&gt;
    - Fixed hash errors when remote source mapped to local file system&lt;/p&gt;&lt;/div&gt;</summary></entry></feed>