Steve Lloyd's ATLAS Grid Tests

Upgrade to AtlasSetup

2010-10-29T15:06:00.002Z

I have changed the setup for my ATLAS jobs so it uses AtlasSetup (rather than AtlasLogin). The magic lines are:


RELEASE=xx.xx.xx
source $VO_ATLAS_SW_DIR/software/$RELEASE/cmtsite/asetup.sh AtlasOffline $RELEASE

VO_ATLAS_SW_DIR is set up automatically and you have to set RELEASE yourself. Since AtlasSetup is only available from Release 16 onwards, jobs going to sites without Release 16 will fail.

Steve's pages update

2010-07-23T09:01:00.002Z

I have done some much needed maintenance and the gstat information is available again (from gstat2). There is also a new page giving the history of the ATLAS Hammercloud tests status http://pprc.qmul.ac.uk/~lloyd/gridpp/hammercloud.html.

ATLAS Tests now running natively on SL5

2010-01-26T10:09:00.001Z

My ATLAS tests (and 'UK Tests') are now running in native SL5 mode on all sites where ATLAS release 15.6.3 (or above) is installed.

UK Tests Working Again

2009-11-04T12:42:00.003Z

I've finally managed to get the UK tests working again. There were several problems that I won't go into. I have tried to make it a bit more robust and now the tests should start using the latest ATLAS release as soon as more than 50% of UK sites have it installed.

ATLAS Release 15

2009-07-08T18:51:00.002Z

Today I finally got round to switching my ATLAS tests to using Release 15. I tried as soon as it came out but the tests all failed and so they did today (as you might have seen) until I finally fixed it this evening. I don't understand why it wouldn't work - I had to modify PYTHONPATH and LD_LIBRARY_PATH by hand eventually.

SE SAM Tests removed

2009-02-24T15:44:00.001Z

I have remove the 'SE' SAM tests as they are no longer run and are superceded by the 'SRMv2' ones.

SAM Updates

2008-12-02T09:35:00.002Z

I have changed the main SAM tests from SRM to SRMv2. I have also added ATLAS specific SAM tests (to join CMS and LHCb ones). It is not clear if these are the right tests. Each SAM summary page now has a list of the tests being polled under the results table. If the experiments want different tests please let me know.

Proxy problems

2008-10-08T11:19:00.002Z

Most tests have been red for the last 24 hrs or so due to failure to submit any jobs. This was due to trying to use an ATLAS production role which seems to stop anything working. The tests should be OK again now I've stopped trying to do this.

New Server

2008-10-01T13:13:00.001Z

I have moved everything on to one dedicated server. There should be no noticeable change but if there is please let me know.

CMS SAM Tests added

2008-09-16T10:51:00.001Z

I've added CMS specific SAM tests to the suite. See
http://pprc.qmul.ac.uk/~lloyd/gridpp/cms_samtest.html.

Network Tests Updated

2008-08-01T11:53:00.000Z

I have revamped the network tests as agreed at a dteam meeting a couple of weeks ago. The tests now involve copying files from the Tier-1 to the local SE, copying the files from the local SE to the WN and copying them back to all SEs. There is quite a high failure rate. Some of this is transfer/catalogue/info failures and some of it is attempts to get more reliable results by using different timings and asking for consistency between them. Sometimes the times vary wildly with smaller files taking longer etc. I am sure there is more to be done.

LHCb Tests

2008-07-22T14:02:00.002Z

I have added LHCb tests to the collection. These are actually SAM tests tailored to LHCb. They are kept separate from the normal (Ops Critical) SAM tests to avoid distorting the availability/reliability statistics. See http://pprc.qmul.ac.uk/~lloyd/gridpp/lhcb_samtest.html.

Local problems

2008-07-21T14:59:00.002Z

The bad news: The raid array I store my outputs and log files on developed an error over the weekend. It is being rebuilt now but may take some time before it comes back.

The good news: I've instigated a 'heartbeat' system so that if all the tests stop running and the pages look green there is something to tell you that in fact it hasn't been updated for many hours.

Network Tests

2008-07-02T09:41:00.002Z

I have introduced some network tests. Test jobs are sent to every UK site once an hour which attempt to copy a number of differently sized test files (1MB, 5MB, 10MB, 50MB and 100MB) from the Storage Element at each of the sites. The transfer times are then used to calculate the average throughput in MB/s. The results are averaged over the last 24 hours to produce a matrix of site to site transfer throughputs. There is an average over the whole matrix on the main UK Grid page. See http://pprc.qmul.ac.uk/~lloyd/gridpp/nettest.html.

New test

2008-06-23T11:35:00.000Z

I have enabled a new test which sends an ATLAS analysis job to ".ac.uk" every 10 minutes and wherever it ends up reads data from the local SE and does some analysis. This should give a much better estimate of success rate as seen by users (as jobs won't go to sites that are down). There is still the problem for users whose data is on the site that is down of course). See http://pprc.qmul.ac.uk/~lloyd/gridpp/uktest.html for a summary.

New views

2008-06-18T15:34:00.000Z

There are a new set of results pages showing all the results for a particular site or Tier-2 on one page. Go to the main page
http://pprc.qmul.ac.uk/~lloyd/gridpp/ukgrid.html and click on one of the institute names down the left hand side. Or click on a Tier-2 just above the big table.

FCR History

2008-06-17T13:20:00.002Z

I have been collecting the FCR history since just before Christmas. I have now put up a new page that shows the percentage of time each site is blacklisted by each VO using the FCR. This can be found at http://pprc.qmul.ac.uk/~lloyd/gridpp/fcrtest.html and is linked from the other test pages as "FCR".

Late Spring Clean

2008-06-13T10:52:00.002Z

I've done some tidying up so that the ATLAS tests are more resilient to changing versions of the ATLAS code and this seems to mean that sites with only Release 13 and not 14 are passing again. I have also tried to make all the times consistent - some were previously GMT and some BST. They are now all supposed to be BST in Summer and GMT in Winter.

Release 14 Complete

2008-06-06T14:13:00.002Z

All three ATLAS tests are now running on ATLAS Release 14. If you want to pass you need to install 14.1.0 or higher.

Release 14 User Analysis Success

2008-06-03T16:09:00.001Z

Release 14 User Analysis seems to work at some sites at least. The Z mass has gone up from 96.661 to 97.203!

Upgrading to ATLAS Release 14

2008-06-03T14:50:00.003Z

I am currently upgrading my tests to ATLAS Release 14 starting with User Analysis. All sites without Release 14, which requires SL4, will fail the test. All sites will probably fail while I get the bugs out on my side! For the time being the other tests will use Release 13 (even if 14 is installed).

New SE Tests

2008-01-22T15:38:00.000Z

There are now some new tests of the UK SEs. These run every hour (for the moment) and attempt to copy a (2.8MB) file to an SE (lcg-cr), check it is there (lcg-lr), read it back (lcg-cp), delete it (lcg-del) and check it is no longer there (lcg-lr). The log files from these operations are available by clicking on the test result. You can see that the results are not 100% correlated with the SAM test results (which is why I did this). See:
http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/setest.html.

Auto RB

2008-01-14T12:40:00.000Z

On the basis of my RB tests I now define an 'Auto RB' which is one of the 'Good' RBs or failing that one of the 'Fair' ones. This RB is then used for my ATLAS tests. This should avoid having to manually switch RBs every time the RAL ones go down.

Major Disruption

2007-12-19T15:44:00.000Z

There was major disruption over the last couple of days after a raid array died and various things had to be recovered from backup. Everything should be running again now but there is some history missing between 13-19 Dec. This could be recovered but is probably not worth it. The system is 'at risk' from now till 3 Jan (no 24x7 here!). Merry Christmas and a Happy New Year to all who use my test results.

ATLAS Tests now all use release 13.0.30

2007-12-06T11:43:00.000Z

I have finally managed to create some 13.0.30 AOD and upload it to (most) UK SEs. All the tests are now using 13.0.30. QMUL, UCL_CCC, Lancaster and Edinburgh are not getting the analysis job at the moment as I could not make the replica at these sites.

Problems at RAL

2007-12-04T12:24:00.000Z

All the RAL RBs seem to be 'stuck' and the lfc is not reachable. I have switched to the ScotGrid RB to try and get the ATLAS tests running again.

Many Changes

2007-10-29T13:28:00.000Z

So finally things have stabilized. The ATLAS and RB tests are running properly again using a database (rather than text files) to improve reliability. The php scripts that provided the summaries have been replaced by static pages to stop the webserver errors. In addition there are now tests of the UK top level BDIIs: http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/bdiitest.html All the tests are summarised on one page (accessible from the GridPP home page): http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/ukgrid.html Plots of SAM performance are also available:
http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/samplots.html

Mnay problems and possible solutions

2007-09-06T09:32:00.000Z

There have been many problems over the summer:
RB Tests: I still cannot submit jobs through IC or Glasgow RBs due to atlas/dteam conflicts. There have been other problems due to a build up of jobs, problems updating my (flat) database of results, an overloaded server etc.
ATLAS Tests: These died on 17 August when there was a power cut and have only recently been revieved. They are OK at the moment.
SAM Tests: There was a problem in caculating 'availability' but this only affected the history (for Tier-1?) and has been fixed.

I am rewritting some of the code and replacing my flat database with sqlite. This has been done for the RB Tests and seems to be OK. There are a few more tweaks to do then I will move the ATLAS tests to the same framework. This should make it more robust.

dteam Woes

2007-07-06T11:28:00.000Z

In order to de-ATLAS my test jobs (especially for the RB tests) I joined the dteam. This was a disaster to start with as my ATLAS jobs failed at some sites due to VOMS/Gridmap file issues and bugs in edg-job-submit. These seem to more or less resolved now except that all the jobs I submit through the Glasgow RB never run. They get submitted to a CE but stay in the queue for ever. No-one seems to know why.

RB Tests

2007-07-06T11:24:00.000Z

There is a new set of tests targeted at UK Resource Brokers. These send (non ATLAS) "Hello World" jobs to each UK RB every 10 minutes to execute on any UK CE. See
http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/rbtest.html.

RB Logging Problems

2007-06-19T15:58:00.000Z

Although I switched to lcgrb02 the logging was still going to lcgrb01. This is because it's hardwired into /opt/edg/etc/edg_wl_ui_cmd_var.conf and can't be overwritten by your own conf file (how dumb is that?). Commented out LoggingDestination and it should be OK now. Stephen B reminds me that it is written here:
http://www.gridpp.ac.uk/deployment/users/faq.html#chooserb

ATLAS Release 13 and replica problems

2007-06-18T15:21:00.000Z

There have been a few problems in the last few days. Firstly some sites installed ATLAS release 13.0.10 which broke some of the tests. At the moment the tests are only running on version 12.0.6 even if 13.0.10 is installed. Secondly something has happened to the UI at QMUL in that I cannot find out about my file replicas any more. This prevented analysis jobs being submitted over the weekend. This is being worked around at the moment by using a fixed list and not trying to obtain replica information dynamically.

Several problems

2007-05-22T11:44:00.000Z

On Saturday morning (19 May) the machine that submits the test jobs crashed. On Sunday the system was moved to a different machine but now ~50% jobs fail to complete. This appears to be correlated with them going through lcgrb02.gridpp.rl.ac.uk rather than lcgrb01.gridpp.rl.ac.uk and looks like an RB problem rather than a problem at my end. I have now switched to the IC RB to see if this improves things.

More hangups

2007-05-11T14:37:00.000Z

System hung up again trying to retrieve the job output. I have changed the scrip so that this is now done in a separate thread which is killed after n minutes so as not to hang the whole script.

Still messed up

2007-05-01T14:23:00.000Z

Things are still messed up. The script seems to hang trying to retrieve output. I have made a slight change and suspended submission until I clear the backlog.

Something messed up this weekend

2007-04-30T08:27:00.000Z

Over the weekend the system got messed up. Job submission seemed to be OK and the jobs ran OK but my scripts kept hanging getting the output back. No idea why. It seems to be OK again now.

Slight bug in list of logs fixed

2007-04-27T09:36:00.000Z

There was a small bug in the php script for the web page causing some jobs not to appear in the filtered list of job on the "All Logs" page although they did appear on the main page. This is now fixed. (Stupid strpos returns an integer that can be 0 if there is a match at the first character but false if there is no match. You have to use === or !=== to test it)

Liverpool replica disappeared again

2007-03-29T09:37:00.000Z

The replica at Liverpool disappeared again but was still in the catalogue. We do not know why. Carl has remade it now.

Adjust times for Summer Time

2007-03-28T10:54:00.000Z

All timestamps appear to be in GMT so changed the web page so that the time of the last job submitted takes summer time into account properly. There are some internal conversions that still need to be checked.

Replicas back

2007-03-28T07:57:00.001Z

The RAL catalogue seems to be fixed now and analysis jobs are running again.

Catalogue broken?

2007-03-27T07:19:00.001Z

I cannot find out any information about any of my replicas. However apparently the replica on disk at Glasgow (at least) is OK. Have raised a GGUS ticket.

Summary Problem

2007-03-16T12:11:00.000Z

There was a problem with the summary csv file having a few entries with only one event. This was due to entries in the original log file being out of time sequence. The file is now sorted before the summary is made.

Hung up

2007-03-16T12:10:00.000Z

The system was hung up this morning. Although jobs were being submitted overnight my script couldn't find their status or get there output. I did it by hand and it was OK. Perhaps there was a temporary RB problem.

A Sheffield Replica Finally

2007-03-15T10:52:00.000Z

Finally managed to make a replica at Sheffield!

Bug fixed

2007-03-15T09:52:00.000Z

The submission system was hung up this morning. This was due to a job being in a "Waiting" state exposing a bug in the script and causing it to abort. This is hopefully now fixed and the backlog has been cleared.

Filter Logs on Status

2007-03-14T17:49:00.001Z

Added the option to filter all log files on the job status, i.e Failed, Aborted, Not successful etc.

Old Logs Available

2007-03-08T09:15:00.001Z

There is a new link "All Logs" which allows one to look at old log and output files. These can be filtered by institute. It's a bit flaky because the webserver isn't really up to it. Logs before February 8 have been lost.

Update RB config file

2007-03-05T09:46:00.000Z

In my atlas.config file change


NSAddresses = "lcgrb02.gridpp.rl.ac.uk:7772";
LBAddresses = "lcgrb02.gridpp.rl.ac.uk:9000";


NSAddresses = {"lcgrb01.gridpp.rl.ac.uk:7772","lcgrb02.gridpp.rl.ac.uk:7772"};
LBAddresses = {{"lcgrb01.gridpp.rl.ac.uk:9000"},{"lcgrb02.gridpp.rl.ac.uk:9000"}};

to try and use load balancing.

Working again

2007-03-04T20:07:00.001Z

Commented out LoggingDestination line in heppc009:/opt/edg/etc/edg_wl_ui_cmd_var.conf Everything seems to be OK again now.

RAL RB Broken

2007-03-02T23:37:00.000Z

All job submission fails:


Selected Virtual Organisation name (from --config-vo option): atlas
Connecting to host lcgrb02.gridpp.rl.ac.uk, port 7772
Logging to host lcgrb01.gridpp.rl.ac.uk, port 9002
**** Error: API_NATIVE_ERROR ****  
Error while calling the "edg_wll_RegisterJobSync" native api 
Unable to Register the Job:
https://lcgrb02.gridpp.rl.ac.uk:9000/rdV_Ep9fG_oqlBypG--UBQ
to the LB logger at: lcgrb01.gridpp.rl.ac.uk:9002
No route to host (edg_wll_ssl_connect())

Why does it try and use lcgrb01.gridpp.rl.ac.uk when I have this in my conf file:


[
VirtualOrganisation = "atlas";
NSAddresses = "lcgrb02.gridpp.rl.ac.uk:7772";
LBAddresses = "lcgrb02.gridpp.rl.ac.uk:9000";

Manchester ce01 Off

2007-03-02T14:11:00.000Z

There are no suitable queues on Manchester ce01 and all my jobs fail so I've switched it off for the time being. ce02 is OK.

Splitting Manchester

2007-03-01T16:50:00.000Z

On Alessandra's request I am splitting Manchester into two - ce01 and ce02 reading from dcache01 and dcache02 respectively. At the moment it isn't quite working because of a bdii problem somewhere.

Minor problem

2007-03-01T16:47:00.000Z

Attempts to make everything fully automatic by killing old processes and deleting their lock files failed because of a bug. Everything stopped overnight. Hopefully now OK.

Proxy expired

2007-02-27T23:39:00.000Z

All job submission stopped this afternoon because my proxy expired. Now running again.

Cleanup not quite right

2007-02-26T09:17:00.000Z

The cleanup didn't work quite right as it cleared the entries out of the log file but not out of the if file and so they kept coming back. An attempt to fix this yesterday failed because of a typo. Trying again.

Automatic cleanup

2007-02-23T12:38:00.000Z

Have introduced some automatic cleanup via the cron job. Scheduled, Ready, Running, Waiting and "Job proxy is expired" jobs are removed from the log after 12 hours. The logs are archived after 7 days. Lock files are removed after 60 minutes and an attempt is made to kill the running process.

OK Now

2007-02-23T10:26:00.000Z

Seems to be OK now. Have put the archiving of the log files in the cron job to run once a day.

Web page not updated

2007-02-23T09:20:00.000Z

The web page has not been updated since 4.30 this morning. The previous lock file was not removed for some reason. Deleted it by hand. Job submission seems to have been OK.

Jobs restarted

2007-02-22T16:59:00.000Z

Cleaned out all pending jobs and restarted submission.

Problem understood

2007-02-22T10:45:00.000Z

The file system could not cope with me storing all the job outputs in one place. The code is being changed so that each day's worth of jobs is in a separate directory. Job submission will be restarted once this has been debugged.

Strange State

2007-02-22T09:28:00.000Z

Not sure what is going on. Summary is all yellow but individual jobs look OK. It appears atest isn't getting the status properly. Stopped submission till I sort it out.

Sheffield replica failed again

2007-02-21T16:59:00.000Z


[lloyd@heppc005 atest]$ export LFC_HOST=lfc.gridpp.rl.ac.uk
[lloyd@heppc005 atest]$ lcg-rep -v -t 120 --vo atlas -d lcgse1.shef.ac.uk 
srm://epgse1.ph.bham.ac.uk/dpm/ph.bham.ac.uk/home/atlas/generated/2007-01-19/
filef607aa8a-5ee3-44a8-bf0f-f0ccbb73aecb
Using grid catalog type: lfc
Using grid catalog : lfc.gridpp.rl.ac.uk
lcg_rep: Permission denied

Looking Good

2007-02-21T09:24:00.000Z

Everything looking much better today. Changed the kill time back to 8 hours.

Status

2007-02-20T22:45:00.000Z

Looking OK at the moment - 70% success

System restarted

2007-02-20T16:55:00.000Z

Cleaned out all remaining jobs at IC from my system and restarted everything using the 2nd RAL RB.

Imperial RB is very slow

2007-02-20T14:12:00.000Z

The Imperial RC is very slow and there is a big backlog of jobs still to be processed ('waiting'). I have stopped job submission and switched the RB to lcgrb02. I have also extended the cancellation time from 8 to 20 hours till the backlog is cleared.