Steve Lloyd's ATLAS Grid Tests

UK Tests Working Again

noreply@blogger.com (Steve Lloyd) — Wed, 04 Nov 2009 12:42:00 +0000

I've finally managed to get the UK tests working again. There were several problems that I won't go into. I have tried to make it a bit more robust and now the tests should start using the latest ATLAS release as soon as more than 50% of UK sites have it installed.

ATLAS Release 15

noreply@blogger.com (Steve Lloyd) — Wed, 08 Jul 2009 18:51:00 +0000

Today I finally got round to switching my ATLAS tests to using Release 15. I tried as soon as it came out but the tests all failed and so they did today (as you might have seen) until I finally fixed it this evening. I don't understand why it wouldn't work - I had to modify PYTHONPATH and LD_LIBRARY_PATH by hand eventually.

SE SAM Tests removed

noreply@blogger.com (Steve Lloyd) — Tue, 24 Feb 2009 15:44:00 +0000

I have remove the 'SE' SAM tests as they are no longer run and are superceded by the 'SRMv2' ones.

SAM Updates

noreply@blogger.com (Steve Lloyd) — Tue, 02 Dec 2008 09:35:00 +0000

I have changed the main SAM tests from SRM to SRMv2. I have also added ATLAS specific SAM tests (to join CMS and LHCb ones). It is not clear if these are the right tests. Each SAM summary page now has a list of the tests being polled under the results table. If the experiments want different tests please let me know.

Proxy problems

noreply@blogger.com (Steve Lloyd) — Wed, 08 Oct 2008 11:19:00 +0000

Most tests have been red for the last 24 hrs or so due to failure to submit any jobs. This was due to trying to use an ATLAS production role which seems to stop anything working. The tests should be OK again now I've stopped trying to do this.

New Server

noreply@blogger.com (Steve Lloyd) — Wed, 01 Oct 2008 13:13:00 +0000

I have moved everything on to one dedicated server. There should be no noticeable change but if there is please let me know.

CMS SAM Tests added

noreply@blogger.com (Steve Lloyd) — Tue, 16 Sep 2008 10:51:00 +0000

I've added CMS specific SAM tests to the suite. See
http://pprc.qmul.ac.uk/~lloyd/gridpp/cms_samtest.html.

Network Tests Updated

noreply@blogger.com (Steve Lloyd) — Fri, 01 Aug 2008 11:53:00 +0000

I have revamped the network tests as agreed at a dteam meeting a couple of weeks ago. The tests now involve copying files from the Tier-1 to the local SE, copying the files from the local SE to the WN and copying them back to all SEs. There is quite a high failure rate. Some of this is transfer/catalogue/info failures and some of it is attempts to get more reliable results by using different timings and asking for consistency between them. Sometimes the times vary wildly with smaller files taking longer etc. I am sure there is more to be done.

LHCb Tests

noreply@blogger.com (Steve Lloyd) — Tue, 22 Jul 2008 14:02:00 +0000

I have added LHCb tests to the collection. These are actually SAM tests tailored to LHCb. They are kept separate from the normal (Ops Critical) SAM tests to avoid distorting the availability/reliability statistics. See http://pprc.qmul.ac.uk/~lloyd/gridpp/lhcb_samtest.html.

Local problems

noreply@blogger.com (Steve Lloyd) — Mon, 21 Jul 2008 14:59:00 +0000

The bad news: The raid array I store my outputs and log files on developed an error over the weekend. It is being rebuilt now but may take some time before it comes back.

The good news: I've instigated a 'heartbeat' system so that if all the tests stop running and the pages look green there is something to tell you that in fact it hasn't been updated for many hours.

Network Tests

noreply@blogger.com (Steve Lloyd) — Wed, 02 Jul 2008 09:41:00 +0000

I have introduced some network tests. Test jobs are sent to every UK site once an hour which attempt to copy a number of differently sized test files (1MB, 5MB, 10MB, 50MB and 100MB) from the Storage Element at each of the sites. The transfer times are then used to calculate the average throughput in MB/s. The results are averaged over the last 24 hours to produce a matrix of site to site transfer throughputs. There is an average over the whole matrix on the main UK Grid page. See http://pprc.qmul.ac.uk/~lloyd/gridpp/nettest.html.

New test

noreply@blogger.com (Steve Lloyd) — Mon, 23 Jun 2008 11:35:00 +0000

I have enabled a new test which sends an ATLAS analysis job to ".ac.uk" every 10 minutes and wherever it ends up reads data from the local SE and does some analysis. This should give a much better estimate of success rate as seen by users (as jobs won't go to sites that are down). There is still the problem for users whose data is on the site that is down of course). See http://pprc.qmul.ac.uk/~lloyd/gridpp/uktest.html for a summary.

New views

noreply@blogger.com (Steve Lloyd) — Wed, 18 Jun 2008 15:34:00 +0000

There are a new set of results pages showing all the results for a particular site or Tier-2 on one page. Go to the main page
http://pprc.qmul.ac.uk/~lloyd/gridpp/ukgrid.html and click on one of the institute names down the left hand side. Or click on a Tier-2 just above the big table.

FCR History

noreply@blogger.com (Steve Lloyd) — Tue, 17 Jun 2008 13:20:00 +0000

I have been collecting the FCR history since just before Christmas. I have now put up a new page that shows the percentage of time each site is blacklisted by each VO using the FCR. This can be found at http://pprc.qmul.ac.uk/~lloyd/gridpp/fcrtest.html and is linked from the other test pages as "FCR".

Late Spring Clean

noreply@blogger.com (Steve Lloyd) — Fri, 13 Jun 2008 10:52:00 +0000

I've done some tidying up so that the ATLAS tests are more resilient to changing versions of the ATLAS code and this seems to mean that sites with only Release 13 and not 14 are passing again. I have also tried to make all the times consistent - some were previously GMT and some BST. They are now all supposed to be BST in Summer and GMT in Winter.

Release 14 Complete

noreply@blogger.com (Steve Lloyd) — Fri, 06 Jun 2008 14:13:00 +0000

All three ATLAS tests are now running on ATLAS Release 14. If you want to pass you need to install 14.1.0 or higher.

Release 14 User Analysis Success

noreply@blogger.com (Steve Lloyd) — Tue, 03 Jun 2008 16:09:00 +0000

Release 14 User Analysis seems to work at some sites at least. The Z mass has gone up from 96.661 to 97.203!

Upgrading to ATLAS Release 14

noreply@blogger.com (Steve Lloyd) — Tue, 03 Jun 2008 14:50:00 +0000

I am currently upgrading my tests to ATLAS Release 14 starting with User Analysis. All sites without Release 14, which requires SL4, will fail the test. All sites will probably fail while I get the bugs out on my side! For the time being the other tests will use Release 13 (even if 14 is installed).

New SE Tests

noreply@blogger.com (Steve Lloyd) — Tue, 22 Jan 2008 15:38:00 +0000

There are now some new tests of the UK SEs. These run every hour (for the moment) and attempt to copy a (2.8MB) file to an SE (lcg-cr), check it is there (lcg-lr), read it back (lcg-cp), delete it (lcg-del) and check it is no longer there (lcg-lr). The log files from these operations are available by clicking on the test result. You can see that the results are not 100% correlated with the SAM test results (which is why I did this). See:
http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/setest.html.

Auto RB

noreply@blogger.com (Steve Lloyd) — Mon, 14 Jan 2008 12:40:00 +0000

On the basis of my RB tests I now define an 'Auto RB' which is one of the 'Good' RBs or failing that one of the 'Fair' ones. This RB is then used for my ATLAS tests. This should avoid having to manually switch RBs every time the RAL ones go down.

Major Disruption

noreply@blogger.com (Steve Lloyd) — Wed, 19 Dec 2007 15:44:00 +0000

There was major disruption over the last couple of days after a raid array died and various things had to be recovered from backup. Everything should be running again now but there is some history missing between 13-19 Dec. This could be recovered but is probably not worth it. The system is 'at risk' from now till 3 Jan (no 24x7 here!). Merry Christmas and a Happy New Year to all who use my test results.

ATLAS Tests now all use release 13.0.30

noreply@blogger.com (Steve Lloyd) — Thu, 06 Dec 2007 11:43:00 +0000

I have finally managed to create some 13.0.30 AOD and upload it to (most) UK SEs. All the tests are now using 13.0.30. QMUL, UCL_CCC, Lancaster and Edinburgh are not getting the analysis job at the moment as I could not make the replica at these sites.

Problems at RAL

noreply@blogger.com (Steve Lloyd) — Tue, 04 Dec 2007 12:24:00 +0000

All the RAL RBs seem to be 'stuck' and the lfc is not reachable. I have switched to the ScotGrid RB to try and get the ATLAS tests running again.

Many Changes

noreply@blogger.com (Steve Lloyd) — Mon, 29 Oct 2007 13:28:00 +0000

So finally things have stabilized. The ATLAS and RB tests are running properly again using a database (rather than text files) to improve reliability. The php scripts that provided the summaries have been replaced by static pages to stop the webserver errors. In addition there are now tests of the UK top level BDIIs: http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/bdiitest.html All the tests are summarised on one page (accessible from the GridPP home page): http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/ukgrid.html Plots of SAM performance are also available:
http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/samplots.html

Mnay problems and possible solutions

noreply@blogger.com (Steve Lloyd) — Thu, 06 Sep 2007 09:32:00 +0000

There have been many problems over the summer:
RB Tests: I still cannot submit jobs through IC or Glasgow RBs due to atlas/dteam conflicts. There have been other problems due to a build up of jobs, problems updating my (flat) database of results, an overloaded server etc.
ATLAS Tests: These died on 17 August when there was a power cut and have only recently been revieved. They are OK at the moment.
SAM Tests: There was a problem in caculating 'availability' but this only affected the history (for Tier-1?) and has been fixed.

I am rewritting some of the code and replacing my flat database with sqlite. This has been done for the RB Tests and seems to be OK. There are a few more tweaks to do then I will move the ATLAS tests to the same framework. This should make it more robust.