Recent changes to Project HipMer

#15 "ERROR: upcrun subprocess terminated with exit code -4" for test_hiper.sh

2022-11-01T01:38:22.869000Z

Hello, Apologies for the late reponse. This ticket arrived at the same time as an update from another and I didn't notice it.

The icc_run.log has this error:
*** Caught a fatal signal (proc 28): SIGILL(4)

Which, in my experience, only happens when you build on 1 machine and execute on a different one. Your build platform must have the same CPU as the execute platform or else you run the risk that some optimization enabled in the build architechure cannot run on the execute architechure.

So I suspect that if you execute "lscpu" on the two environments, you will see different chipsets.

Maybe you can build on the execution platform? or somehow configure cmake to cross compile?

-Rob

#14 Issue while executing test_hipmer.sh

2022-11-01T01:25:59.355000Z

Hi sorry for the late reply. but this was a temporary problem that NERSC fixed a few weeks ago.

"ERROR: upcrun subprocess terminated with exit code -4" for test_hiper.sh

2022-10-14T12:15:18.380000Z

Hi ,

I compiled HipMer using Intel compilers and OpenMPI-4.1.1. The build is carried out fine but I am getting the following issue while execution of "test_hipmer.sh"

*** Caught a fatal signal (proc 29): SIGILL(4)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace.
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace.
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.

ERROR: upcrun subprocess terminated with exit code -4
Check run.log for errors
Assembly failed

Auto-restart is disabled. Aborting at stage: meraculous-21
Missing final assembly: results/final_assembly.fa

FAILED run at 2022-10-14 11:21:01.917376 in 43.70 s

I am attaching the run log file related to the issue along with HipMer and UPC build logs.

Can you please let me know how do I get rid of this issue?

#14 Issue while executing test_hipmer.sh

2022-10-14T10:15:24.958000Z

I tried to navigate through the link you mentioned but this is the issue:
Sorry, there was an error downloading /home/r/regan/www/HipMer:
can't log in as wwwhpss

#14 Issue while executing test_hipmer.sh

2022-10-13T17:13:38.086000Z

Hello NERSC is still working on a fix but set up a temporary solution so that you can proceed and download the test data from:
https://portal-dev.nersc.gov/archive/home/r/regan/www/HipMer
The easiest hack would be to edit hipmer_setup_ecoli_data.sh
to use portal-dev.nersc.gov in the HIPMER_ECOLI_DATA_URL variable.

#14 Issue while executing test_hipmer.sh

2022-10-11T16:49:41.065000Z

This is hosted at NERSC, and this happens a few hours per week when NERSC's tape system is down for maintenance, so generally waiting a few hours fixes it.
HOWEVER today, there is a real problem with this service and I have opened a ticket with NERSC to have them fix it.

#14 Issue while executing test_hipmer.sh

2022-10-11T08:32:32.561000Z

Hi Rob,

I am trying ecoli, chr14 datasets and I am getting the below error while executing test_hipmer.sh script:

HIPMER_VERSION: 1.2.3
+ Started at Tue Oct 11 11:42:31 IST 2022 in 0 s: ./test_hipmer.sh ecoli ( pid:144392)
+ Linking latest_run to output in /home/sushma/hipmer/1.2.3/ompi411/gcc920/bin/ecoli-32--144392_20221011_114231
+ Attempting to copy/download the data for ecoli
Discovered base path of /home/sushma/hipmer/1.2.3/ompi411/gcc920/bin
Setting up /home/sushma/hipmer/1.2.3/ompi411/gcc920/bin/ecoli-32-144392_20221011_114231 with ecoli data set at Tue Oct 11 11:42:31 IST 2022

*Downloading ecoli fastq from the NERSC HPSS system... this could take a while
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 528 100 528 0 0 540 0 --:--:-- --:--:-- --:--:-- 540
Warning: Transient problem: HTTP error Will retry in 120 seconds. 3 retries
Warning: left.
100 528 100 528 0 0 404 0 0:00:01 0:00:01 --:--:-- 404
Warning: Transient problem: HTTP error Will retry in 120 seconds. 2 retries
Warning: left.
100 528 100 528 0 0 519 0 0:00:01 0:00:01 --:--:-- 520
Warning: Transient problem: HTTP error Will retry in 120 seconds. 1 retries
Warning: left.
100 528 100 528 0 0 510 0 0:00:01 0:00:01 --:--:-- 510 * gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Download failed, trying again in 30 min

Can I get some help here on executing these datasets?

#14 Issue while executing test_hipmer.sh

2022-08-22T22:07:27.300000Z

Hello Sushma,

Those warnings and errors are benign. The first "error: getstripe failed for" is something we should surpress. It attempted to set the lustre striping of your test directory, but, obviously, you do not have lustre, a parallel filesystem on your single node.

The second warning: "UPC Runtime warning: Requested shared memory (2869 MB) > available (1984 MB) on node 0 (milanlogin): using 1984 MB per thread instead"
Is informing you that HipMer requested 2869MB per rank of shared heap, but only 1984MB per rank were allowed to allocate by your system, so 70% of what we recommend for the size of your system of 500GB per node.

The third warning: "Warning: Could not find the psutil module, memory information and profiling may be inaccurate"
is informing you that python does not have the psutil installed and so some reported metrics may be less accurate.

The fourth warning:
upcrun: WARNING: Given options request 1 nodes for 128 processes,
upcrun: WARNING: but there is no setting for 'smp_spawn_nodes'.
upcrun: WARNING: Will use setting for 'smp_spawn' instead.
upcrun: WARNING: THREAD LAYOUT MIGHT NOT MATCH YOUR REQUEST.

Just means that cores & ranks may not be properly bound which could suffer a minor performance hit.

The important part was that the validation tests passed, so you should be good to go to run HipMer on your single node.

If you have more than one node with a high speed network like infiniband, you will need to do some more work configuring UPC and UPC++ to utilize those. This is out of the scope of my expertise in HipMer support... the install_upc.sh script that we provide is typically good only for single-machine installs, since UPC and UPC++ usually need extra options and networking device library paths to build that support.

Best,
Rob

#14 Issue while executing test_hipmer.sh

2022-08-22T12:44:36.534000Z

Hello Rob,

With the help of the options mentioned and building it clean from scratch, HipMER is built successfully. I am able to execute test_hipmer.sh for various datasets. I am attaching the log of test_hipmer.sh for the argument 'validation'. Please let me know if the error messages printed are expected and if the test result is as expected.

#14 Issue while executing test_hipmer.sh

2022-08-16T18:35:12.620000Z

Hello Sushma,

Thanks for posting the detailed logs and reports. We believe that you have multiple installations of UPC / UPC++ on your system and CMake is getting confused and choosing the one without pshm support to build HipMer.

Specifically in the attached file build-opt_cupc2c-gasnet-config.txt: /home/sushma/hipmer/gcc/new/berkeley_upc-2022.5.0_new/build/install looks like it was properly built using the default: --enable-pshm

But in your file upcrun_main-32_out.txt:
It uses /home//sushma/hipmer/gcc/new/berkeley_upc-2022.5.0/build/install, which specifically disables pshm: --disable-pshm.

We are surprised that you were able to link the application into an executable as it would have core dumped immediately were two versions of gasnet to be used within the same binary.

Please try again with a clean installation or explicitly force hipmer to build with your install that supports pshm: /home/sushma/hipmer/gcc/new/berkeley_upc-2022.5.0_new/build/install

The build process picks up upcc and upcxx by various environmental variables (PATH, CMAKE_* etc ) and you might be able to override it with CMAKE_UPC_COMPILER, but the best thing to do is to clear out or rename the path to the old build(s) so that CMake does not pick them up.

-Rob