You can subscribe to this list here.
| 2003 |
Jan
|
Feb
(4) |
Mar
(5) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
(8) |
Nov
(1) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2004 |
Jan
(3) |
Feb
(4) |
Mar
(6) |
Apr
(1) |
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
(1) |
Nov
|
Dec
|
|
From: C. R. D. <rya...@us...> - 2004-10-21 18:01:40
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv28830 Modified Files: excel_searches.pl Log Message: Tweaked the parser so as not to deem "malformed" records that specify port 8080 instead of just 80 Index: excel_searches.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_searches.pl,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -r1.7 -r1.8 *** excel_searches.pl 13 Feb 2004 01:18:35 -0000 1.7 --- excel_searches.pl 21 Oct 2004 18:01:18 -0000 1.8 *************** *** 62,66 **** $time = $5; if ( !&IsFilterEntry( $logEntry ) && ( /q\=\S+/ ) && !( /hist\=true/ ) ) { ! if ( /dlese\.org\s+80\s+search\s+(\d+)\s+(\d+)\s+(\-?\d+)[^\"]\".*\"<qle>(.*)<\/qle>\"/ ) { $numResults = $1; $totalResources = $2; --- 62,66 ---- $time = $5; if ( !&IsFilterEntry( $logEntry ) && ( /q\=\S+/ ) && !( /hist\=true/ ) ) { ! if ( /dlese\.org\s+[80]+\s+search\s+(\d+)\s+(\d+)\s+(\-?\d+)[^\"]\".*\"<qle>(.*)<\/qle>\"/ ) { $numResults = $1; $totalResources = $2; |
|
From: C. R. D. <rya...@us...> - 2004-09-29 19:15:40
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv5368 Added Files: excel_searches_no_criteria.pl excel_searches_only_criteria.pl Log Message: A couple new scripts/reports for the quality assessment project, one shows all searches with only criteria specified, and the other shows all searches with no criteria specified. --- NEW FILE --- #!/usr/bin/perl # This script parses the Excel-formatted logfiles that get generated by # excel_searches.pl, and filters out all searches where controlled # vocabulary criteria are specified, producing a report of all keyword- # only searches. Run this against the .filtered reports # # Format for use: # # excel_searches_no_criteria.pl [input_xls] [output_xls] # # Ryan Deardorff, Sep. 2004 $log_in = $ARGV[ 0 ]; $log_out = $ARGV[ 1 ]; open( IN, $log_in ) || die "Couldn't open $log_in"; $firstLine = <IN>; $firstLine =~ s/^([^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]*).*/$1/; while ( <IN> ) { if ( /^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t\"[^\t]*\"(\t.*)$/ ) { if ( $1 !~ /1/ ) { $outData .= $_; } else { $emptyCount++; } } $linesCount++; } close( IN ); open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; print OUT $firstLine . $outData; close( OUT ); print "Processed $linesCount entries, and threw out $emptyCount entries with vocab criteria selected.\n"; exit( 0 ); --- NEW FILE --- #!/usr/bin/perl # This script parses the Excel-formatted logfiles that get generated by # excel_searches.pl, and produces a report of all searches where *only* # vocab criteria were selected, and no keyword was entered. This must # be run over the original .xls search file (not the .filtered) since # empty keywords are of interest in this case. # # Format for use: # # excel_searches_only_criteria.pl [input_xls] [output_xls] # # Ryan Deardorff, Sep. 2004 $log_in = $ARGV[ 0 ]; $log_out = $ARGV[ 1 ]; open( IN, $log_in ) || die "Couldn't open $log_in"; $firstLine = <IN>; while ( <IN> ) { if ( /^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t\"([^\t]*)\"\t1\-\-10(\t.*)$/ ) { $totalSearches++; if ( ( $1 !~ /\S/ ) && ( $2 =~ /1/ ) ) { $outData .= $_; } else { $emptyCount++; } } else { $emptyCount++; } $linesCount++; } close( IN ); open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; print OUT $firstLine . $outData; close( OUT ); print "Processed $linesCount entries, found $totalSearches total searches.\n"; exit( 0 ); |
|
From: C. R. D. <rya...@us...> - 2004-09-21 17:19:14
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv5377 Modified Files: run_log_scripts.pl Log Message: Updated the location of the redirect logs from /rd/ to /rd.dlese.org/ Index: run_log_scripts.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/run_log_scripts.pl,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -r1.3 -r1.4 *** run_log_scripts.pl 7 Oct 2003 23:11:44 -0000 1.3 --- run_log_scripts.pl 21 Sep 2004 17:19:05 -0000 1.4 *************** *** 34,38 **** $DDS_HIST_LOGS = $DDS_SEARCH_LOGS; # "/export/www/webroot/logs/flood/"; Histogram search is now same as regular search... # Location of DDS redirect (resource link-clicking) logs: ! $RD_LOGS = "/export/www/webroot/logs/rd/"; if ( $ARGV[ 0 ] =~ /^(\d\d\d\d\.\d\d)$/ ) { --- 34,38 ---- $DDS_HIST_LOGS = $DDS_SEARCH_LOGS; # "/export/www/webroot/logs/flood/"; Histogram search is now same as regular search... # Location of DDS redirect (resource link-clicking) logs: ! $RD_LOGS = "/export/www/webroot/logs/rd.dlese.org/"; if ( $ARGV[ 0 ] =~ /^(\d\d\d\d\.\d\d)$/ ) { *************** *** 99,103 **** print "Running redirect excel script...\n"; chdir( $EXCEL_DIR ); ! $log = $RD_LOGS . "rd.${month}.log"; if ( -e $log . ".dns" ) { &psystem( "excel_redirects.pl ${log}.dns redirects.${month}.xls" ); --- 99,103 ---- print "Running redirect excel script...\n"; chdir( $EXCEL_DIR ); ! $log = $RD_LOGS . "rd.dlese.org.${month}.log"; if ( -e $log . ".dns" ) { &psystem( "excel_redirects.pl ${log}.dns redirects.${month}.xls" ); |
|
From: <ben...@id...> - 2004-05-22 12:30:30
|
Dear Open Source developer I am doing a research project on "Fun and Software Development" in which I kindly invite you to participate. You will find the online survey under http://fasd.ethz.ch/qsf/. The questionnaire consists of 53 questions and you will need about 15 minutes to complete it. With the FASD project (Fun and Software Development) we want to define the motivational significance of fun when software developers decide to engage in Open Source projects. What is special about our research project is that a similar survey is planned with software developers in commercial firms. This procedure allows the immediate comparison between the involved individuals and the conditions of production of these two development models. Thus we hope to obtain substantial new insights to the phenomenon of Open Source Development. With many thanks for your participation, Benno Luthiger PS: The results of the survey will be published under http://www.isu.unizh.ch/fuehrung/blprojects/FASD/. We have set up the mailing list fa...@we... for this study. Please see http://fasd.ethz.ch/qsf/mailinglist_en.html for registration to this mailing list. _______________________________________________________________________ Benno Luthiger Swiss Federal Institute of Technology Zurich 8092 Zurich Mail: benno.luthiger(at)id.ethz.ch _______________________________________________________________________ |
|
From: C. R. D. <rya...@us...> - 2004-04-01 21:09:12
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv8071 Modified Files: CHANGES.txt excel_histogram.pl Log Message: Updated histogram parser to start using the XML <qle> entries since they are now available (hist search used to use it's own search format prior to DDS v3) Index: CHANGES.txt =================================================================== RCS file: /cvsroot/dlese/logs-project/CHANGES.txt,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -r1.4 -r1.5 *** CHANGES.txt 18 Mar 2004 23:30:17 -0000 1.4 --- CHANGES.txt 1 Apr 2004 20:57:03 -0000 1.5 *************** *** 45,53 **** 2) Added "Collection=ANY" row to the histogram count reports script. ! 3) Fixed a couple bugs that were causing faulty data: ! - Collapsing of certain vocabs was causing innacurate data and positive values for vocabs that don't even display on the public interface (i.e., "To be supplied") - Search .count parser was looking one column too far into the filtered reports, so it was ignoring the first controlled vocab value ("Grade range=DLESE:General public") ! --- 45,55 ---- 2) Added "Collection=ANY" row to the histogram count reports script. ! 3) Fixed a few bugs that were causing faulty data: ! - Collapsing of certain vocabs was causing innacurate data and positive values for vocabs that don't even display on the public interface (i.e., "To be supplied") - Search .count parser was looking one column too far into the filtered reports, so it was ignoring the first controlled vocab value ("Grade range=DLESE:General public") ! - Updated the histogram parser to use the XML-logged data instead of the query string, ! which was all that was available under V2.x of DDS, and which is not as consistent in ! structure as it used to be, so many histogram searches weren't being counted Index: excel_histogram.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_histogram.pl,v retrieving revision 1.8 retrieving revision 1.9 diff -C2 -r1.8 -r1.9 *** excel_histogram.pl 24 Mar 2004 21:26:40 -0000 1.8 --- excel_histogram.pl 1 Apr 2004 20:57:05 -0000 1.9 *************** *** 40,82 **** $resultRangeStart = 0; if ( !&IsFilterEntry( $_ ) && ! ( $request =~ /[\&\?]s\=(\d+)/ ) ) { $resultRangeStart = $1 + 1; # Most humans start counting at 1, not 0 like we do if ( $resultRangeStart == 1 ) { # We only count "new" searches, not "paged" ones ! if ( /\<qle\>(.*)\<\/qle\>/ ) { ! $xmlData = " " . $1 . " "; ! $thisKeywords = ""; ! if ( $xmlData =~ /\<q\>(.*)\<\/q\>/ ) { ! $thisKeywords = $1; } ! while ( $xmlData =~ s/\<([^\>]+)\>([^\<]+)\<\/\1\>// ) { ! if ( ( $1 eq 'type' ) && ( $2 eq 'histogram' ) ) { ! $isHistogram++; } ! if ( $currentVocabHist{ "$1=$2" } ) { ! if ( $1 eq 'ky' ) { ! $thisCollection = $2; ! } ! else { ! $thisCriteria = "$1=$2"; ! } } } - $resultRangeStop = $resultRangeStart + 9; - $collectionLabel = $currentVocab{ 'ky=' . $thisCollection }; - $collectionLabel =~ s/^Collection=//; - if ( $isHistogram ) { - print OUT "$month $day, $year $time\t$currentVocab{ $thisCriteria }\t$collectionLabel\t$ip\n"; - } - $histCount++; } ! else { ! print "Encountered malformed log entry:\n\n" . $_ . "\n\n"; ! $malformed++; } ! } } elsif ( $crawlerIP{ $ip } ) { $crawlers++; ! } } else { --- 40,76 ---- $resultRangeStart = 0; if ( !&IsFilterEntry( $_ ) && ! ( /dlese\.org\s+80\s+search\s+\d+\s+\d+\s+(\-?\d+)[^\"]\".*\"<qle>(.*)<\/qle>\"/ ) ) { $resultRangeStart = $1 + 1; # Most humans start counting at 1, not 0 like we do if ( $resultRangeStart == 1 ) { # We only count "new" searches, not "paged" ones ! $xmlData = " " . $2 . " "; ! $thisKeywords = ""; ! if ( $xmlData =~ /\<q\>(.*)\<\/q\>/ ) { ! $thisKeywords = $1; ! } ! while ( $xmlData =~ s/\<([^\>]+)\>([^\<]+)\<\/\1\>// ) { ! if ( ( $1 eq 'type' ) && ( $2 eq 'histogram' ) ) { ! $isHistogram++; } ! if ( $currentVocabHist{ "$1=$2" } ) { ! if ( $1 eq 'ky' ) { ! $thisCollection = $2; } ! else { ! $thisCriteria = "$1=$2"; } } } ! $resultRangeStop = $resultRangeStart + 9; ! $collectionLabel = $currentVocab{ 'ky=' . $thisCollection }; ! $collectionLabel =~ s/^Collection=//; ! if ( $isHistogram ) { ! print OUT "$month $day, $year $time\t$currentVocab{ $thisCriteria }\t$collectionLabel\t$ip\n"; } ! $histCount++; ! } } elsif ( $crawlerIP{ $ip } ) { $crawlers++; ! } } else { |
|
From: C. R. D. <rya...@us...> - 2004-03-24 21:37:22
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv607 Modified Files: excel_histogram.pl Log Message: Fixed the histogram search parser to use the XML <qle> entries--prior to August histogram searches did not use this, but the newer script had not been updated for it, and was missing some data because it was assuming that the query URL structure was more rigid than it is. Index: excel_histogram.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_histogram.pl,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -r1.7 -r1.8 *** excel_histogram.pl 13 Feb 2004 01:18:35 -0000 1.7 --- excel_histogram.pl 24 Mar 2004 21:26:40 -0000 1.8 *************** *** 11,15 **** # Revision 1: Oct. 2003 for DDS v.3.0.x # Revision 2: Jan. 2004 (filter results > 10) ! # Revision 3: Feb. 2003 (get rid of keywords column) require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; --- 11,16 ---- # Revision 1: Oct. 2003 for DDS v.3.0.x # Revision 2: Jan. 2004 (filter results > 10) ! # Revision 3: Feb. 2004 (get rid of keywords column) ! # Revision 4: Mar. 2004 (convert vocab parser to use <qle> xml instead of query string) require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; *************** *** 24,77 **** open( IN, $log_in ) || die "Couldn't open $log_in"; while ( <IN> ) { ! if ( /query\.do.*[\&\?]hist\=true/i ) { ! if ( /^(\S+)\s+\S+\s+\S+\s+\[(\d+)\/(\w+)\/(\d+)\:([\d\:]+)\s+\S+\]\s+\"\S+\s+([^\s\"]+)[^\"]*\"/ ) { ! $ip = $1; ! $day = $2; ! $month = $3; ! $year = $4; ! $time = $5; ! $request = &Decode( $6 ); ! $request =~ s/\;jsessionid\=[^\&\?\;\s]+//; ! $thisKeywords = ""; ! $thisCriteria = ""; ! $thisCollection = ""; ! $pageRangeInfo = 0; ! $resultRangeStart = 0; ! if ( !&IsFilterEntry( $_ ) && ! ( $request =~ /query\.do.*[\&\?]hist\=true\&(\S+)/ ) ) { ! @theseCriteria = split( /\&/, $1 ); ! foreach $i ( 0..$#theseCriteria ) { ! if ( $theseCriteria[ $i ] =~ /^s\=(\d+)/ ) { ! $resultRangeStart = $1 + 1; # Most humans start counting at 1, not 0 like we do ! } ! elsif ( $theseCriteria[ $i ] =~ /^q\=(.*)/ ) { $thisKeywords = $1; } ! elsif ( $currentVocabHist{ $theseCriteria[ $i ] } ) { ! if ( $theseCriteria[ $i ] =~ /^ky\=(.*)/ ) { ! $thisCollection = $1; } ! else { ! $thisCriteria = $theseCriteria[ $i ]; } } } ! $resultRangeStop = $resultRangeStart + 9; ! $collectionLabel = $currentVocab{ 'ky=' . $thisCollection }; ! $collectionLabel =~ s/^Collection=//; ! if ( $resultRangeStart == 1 ) { ! print OUT "$month $day, $year $time\t$currentVocab{ $thisCriteria }\t$collectionLabel\t$ip\n"; } - $histCount++; - } - elsif ( $crawlerIP{ $ip } ) { - $crawlers++; } } ! else { ! "Encountered malformed log entry:\n\n" . $_ . "\n\n"; ! $malformed++; } } $lines++; } --- 25,87 ---- open( IN, $log_in ) || die "Couldn't open $log_in"; while ( <IN> ) { ! $isHistogram = 0; ! if ( /^(\S+)\s+\S+\s+\S+\s+\[(\d+)\/(\w+)\/(\d+)\:([\d\:]+)\s+\S+\]\s+\"\S+\s+([^\s\"]+)[^\"]*\"/ ) { ! $ip = $1; ! $day = $2; ! $month = $3; ! $year = $4; ! $time = $5; ! $request = &Decode( $6 ); ! $request =~ s/\;jsessionid\=[^\&\?\;\s]+//; ! $thisKeywords = ""; ! $thisCriteria = ""; ! $thisCollection = ""; ! $pageRangeInfo = 0; ! $resultRangeStart = 0; ! if ( !&IsFilterEntry( $_ ) && ! ( $request =~ /[\&\?]s\=(\d+)/ ) ) { ! $resultRangeStart = $1 + 1; # Most humans start counting at 1, not 0 like we do ! if ( $resultRangeStart == 1 ) { # We only count "new" searches, not "paged" ones ! if ( /\<qle\>(.*)\<\/qle\>/ ) { ! $xmlData = " " . $1 . " "; ! $thisKeywords = ""; ! if ( $xmlData =~ /\<q\>(.*)\<\/q\>/ ) { $thisKeywords = $1; } ! while ( $xmlData =~ s/\<([^\>]+)\>([^\<]+)\<\/\1\>// ) { ! if ( ( $1 eq 'type' ) && ( $2 eq 'histogram' ) ) { ! $isHistogram++; } ! if ( $currentVocabHist{ "$1=$2" } ) { ! if ( $1 eq 'ky' ) { ! $thisCollection = $2; ! } ! else { ! $thisCriteria = "$1=$2"; ! } } } + $resultRangeStop = $resultRangeStart + 9; + $collectionLabel = $currentVocab{ 'ky=' . $thisCollection }; + $collectionLabel =~ s/^Collection=//; + if ( $isHistogram ) { + print OUT "$month $day, $year $time\t$currentVocab{ $thisCriteria }\t$collectionLabel\t$ip\n"; + } + $histCount++; } ! else { ! print "Encountered malformed log entry:\n\n" . $_ . "\n\n"; ! $malformed++; } } } ! elsif ( $crawlerIP{ $ip } ) { ! $crawlers++; } } + else { + print "Encountered malformed log entry:\n\n" . $_ . "\n\n"; + $malformed++; + } $lines++; } |
|
From: C. R. D. <rya...@us...> - 2004-03-18 23:40:31
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv4940 Modified Files: CHANGES.txt excel_histogram_count.pl Log Message: Updated V1 (library) scripts to include 0 counts, and added another revision entry in CHANGES.txt Index: CHANGES.txt =================================================================== RCS file: /cvsroot/dlese/logs-project/CHANGES.txt,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -r1.3 -r1.4 *** CHANGES.txt 20 Jan 2004 19:46:24 -0000 1.3 --- CHANGES.txt 18 Mar 2004 23:30:17 -0000 1.4 *************** *** 1,5 **** Log Analysis Script/Report Changes _________________________________________________________________________________________ ! Revision 1 (DDS v3.0.x) Oct. 2003 1) Merging - No changes needed (but redirect logs are now added to the merge). --- 1,5 ---- Log Analysis Script/Report Changes _________________________________________________________________________________________ ! Revision 1 Oct. 2003 1) Merging - No changes needed (but redirect logs are now added to the merge). *************** *** 25,29 **** _________________________________________________________________________________________ ! Revision 2 (DDS v3.0.x) Jan. 2004 1) Excel search reports: --- 25,29 ---- _________________________________________________________________________________________ ! Revision 2 Jan. 2004 1) Excel search reports: *************** *** 34,35 **** --- 34,53 ---- 2) Excel filtered and count reports: - All "paged" searches (results > 10) are now thrown out + + _________________________________________________________________________________________ + Revision 3 Mar. 2004 + + 1) Excel search and browse .count reports updated so that 0 counts are included, which means + traversing the vocab list, so I also added a new array of vocabs to suppress (i.e. "None + of the above", etc.). This means that .count reports should have a consistent number of + rows from month to month, and will be easier to compare/collate. + + 2) Added "Collection=ANY" row to the histogram count reports script. + + 3) Fixed a couple bugs that were causing faulty data: + - Collapsing of certain vocabs was causing innacurate data and positive values for + vocabs that don't even display on the public interface (i.e., "To be supplied") + - Search .count parser was looking one column too far into the filtered reports, so it + was ignoring the first controlled vocab value ("Grade range=DLESE:General public") + + Index: excel_histogram_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_histogram_count.pl,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -r1.5 -r1.6 *** excel_histogram_count.pl 8 Mar 2004 22:26:07 -0000 1.5 --- excel_histogram_count.pl 18 Mar 2004 23:30:18 -0000 1.6 *************** *** 45,49 **** $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; $outData .= "Search criteria\tTotal searches\n"; - #@kz = keys( %currentVocab ); push( @currentVocabLabels, "Collection=ANY" ); @kz = @currentVocabLabels; --- 45,48 ---- |
|
From: C. R. D. <rya...@us...> - 2004-03-18 23:40:11
|
Update of /cvsroot/dlese/logs-project/V1 In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv4940/V1 Modified Files: excel_histogram_count.pl excel_searches.pl excel_searches_count.pl Log Message: Updated V1 (library) scripts to include 0 counts, and added another revision entry in CHANGES.txt Index: excel_histogram_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/V1/excel_histogram_count.pl,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -r1.2 -r1.3 *** excel_histogram_count.pl 23 Feb 2004 20:15:47 -0000 1.2 --- excel_histogram_count.pl 18 Mar 2004 23:30:19 -0000 1.3 *************** *** 11,14 **** --- 11,16 ---- # Ryan Deardorff, Dec. 2002 + require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; + $log_in = $ARGV[ 0 ]; $log_out = $ARGV[ 1 ]; *************** *** 35,41 **** $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; $outData .= "Search criteria\tTotal searches\n"; ! @kz = keys( %criteriaCount ); @kz = sort( @kz ); foreach $i ( 0..$#kz ) { $outData .= "$kz[ $i ]\t$criteriaCount{ $kz[ $i ] }\n"; } --- 37,46 ---- $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; $outData .= "Search criteria\tTotal searches\n"; ! @kz = keys( %currentVocabHist ); @kz = sort( @kz ); foreach $i ( 0..$#kz ) { + if ( !$criteriaCount{ $kz[ $i ] } ) { + $criteriaCount{ $kz[ $i ] } = '0'; + } $outData .= "$kz[ $i ]\t$criteriaCount{ $kz[ $i ] }\n"; } Index: excel_searches.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/V1/excel_searches.pl,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** excel_searches.pl 10 Sep 2003 20:37:45 -0000 1.1 --- excel_searches.pl 18 Mar 2004 23:30:28 -0000 1.2 *************** *** 21,25 **** @key = sort( @key ); foreach $i ( 0..$#key ) { ! print OUT "\t$key[ $i ]"; } print OUT "\n"; --- 21,27 ---- @key = sort( @key ); foreach $i ( 0..$#key ) { ! if ( $key[ $i ] !~ /^topic\=/ ) { # Topic wasn't available as a search in V1 library ! print OUT "\t$key[ $i ]"; ! } } print OUT "\n"; *************** *** 78,86 **** print OUT "$month $day, $year $time\t$ip\t$numResults\t$totalResources\t\"$keywordTerms\"\t$resultRangeStart--$resultRangeStop"; foreach $i ( 0..$#key ) { ! if ( $theseCriteria{ $key[ $i ] } ) { ! print OUT "\t1"; ! } ! else { ! print OUT "\t0"; } } --- 80,90 ---- print OUT "$month $day, $year $time\t$ip\t$numResults\t$totalResources\t\"$keywordTerms\"\t$resultRangeStart--$resultRangeStop"; foreach $i ( 0..$#key ) { ! if ( $key[ $i ] !~ /^topic\=/ ) { # Topic wasn't available as a search in V1 library ! if ( $theseCriteria{ $key[ $i ] } ) { ! print OUT "\t1"; ! } ! else { ! print OUT "\t0"; ! } } } Index: excel_searches_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/V1/excel_searches_count.pl,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -r1.2 -r1.3 *** excel_searches_count.pl 23 Feb 2004 20:15:47 -0000 1.2 --- excel_searches_count.pl 18 Mar 2004 23:30:28 -0000 1.3 *************** *** 11,14 **** --- 11,16 ---- # Ryan Deardorff, Dec. 2002 + require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; + $log_in = $ARGV[ 0 ]; $log_out = $ARGV[ 1 ]; *************** *** 40,43 **** --- 42,48 ---- $criteriaCount{ $criteriaOf[ $i ] }++; } + elsif ( !$criteriaCount{ $criteriaOf[ $i ] } ) { + $criteriaCount{ $criteriaOf[ $i ] } = '0'; + } $i++; } |
|
From: C. R. D. <rya...@us...> - 2004-03-10 00:04:40
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv6700 Modified Files: excel_searches_count.pl merge_logs.pl Log Message: Fixed the search count script so that it doesn't ignore the first column of vocabs (was causing "General public" output row to be lost), and updated the merge script to be able to handle referrer mappings that extend past the domain name (for inclusion of serc.carleton.edu/quantskills) Index: excel_searches_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_searches_count.pl,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -r1.4 -r1.5 *** excel_searches_count.pl 8 Mar 2004 22:47:57 -0000 1.4 --- excel_searches_count.pl 9 Mar 2004 23:39:14 -0000 1.5 *************** *** 18,22 **** open( IN, $log_in ) || die "Couldn't open $log_in"; $firstLine = <IN>; ! if ( $firstLine =~ /^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(.*)$/ ) { $allCriteria = $1; while ( $allCriteria =~ s/^\t([^\t]+)// ) { --- 18,22 ---- open( IN, $log_in ) || die "Couldn't open $log_in"; $firstLine = <IN>; ! if ( $firstLine =~ /^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(.*)$/ ) { $allCriteria = $1; while ( $allCriteria =~ s/^\t([^\t]+)// ) { *************** *** 28,32 **** while ( <IN> ) { ! if ( /^([^\t]+)\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(\t.*)$/ ) { if ( !$firstDate ) { $firstDate = $1; --- 28,32 ---- while ( <IN> ) { ! if ( /^([^\t]+)\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(\t.*)$/ ) { if ( !$firstDate ) { $firstDate = $1; Index: merge_logs.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/merge_logs.pl,v retrieving revision 1.8 retrieving revision 1.9 diff -C2 -r1.8 -r1.9 *** merge_logs.pl 7 Oct 2003 23:11:44 -0000 1.8 --- merge_logs.pl 9 Mar 2004 23:39:14 -0000 1.9 *************** *** 247,256 **** if ( $referrer =~ s/(\/.*)$// ) { $restOfReferrer = $1; } if ( $ReferrerMapping{ "\L$referrer" } ) { ! $referrer = $ReferrerMapping{ $referrer }; ! #print STDERR "Converted: $pf\n\n"; $pf = $one . $http . $referrer . $restOfReferrer . $three; - #print STDERR "To: $pf\n\n"; } } --- 247,262 ---- if ( $referrer =~ s/(\/.*)$// ) { $restOfReferrer = $1; + $baseRefDir = ''; + if ( $restOfReferrer =~ /^(\/[^\/]+)\// ) { + $baseRefDir = $1; + } } if ( $ReferrerMapping{ "\L$referrer" } ) { ! $referrer = $ReferrerMapping{ "\L$referrer" }; ! $pf = $one . $http . $referrer . $restOfReferrer . $three; ! } ! elsif ( $ReferrerMapping{ "\L$referrer$baseRefDir" } ) { # Convert mappings like serc.carleton.edu/quantskills ! $referrer = $ReferrerMapping{ "\L$referrer$baseRefDir" }; $pf = $one . $http . $referrer . $restOfReferrer . $three; } } |
|
From: C. R. D. <rya...@us...> - 2004-03-08 23:12:35
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv11325 Modified Files: excel_searches_count.pl Log Message: Removed debugging print Index: excel_searches_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_searches_count.pl,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -r1.3 -r1.4 *** excel_searches_count.pl 8 Mar 2004 22:26:08 -0000 1.3 --- excel_searches_count.pl 8 Mar 2004 22:47:57 -0000 1.4 *************** *** 43,47 **** } elsif ( !$criteriaCount{ $criteriaOf[ $i ] } && !$noDisplayVocabs{ $criteriaOf[ $i ] } ) { - print $criteriaOf[ $i ] . "\n"; $criteriaCount{ $criteriaOf[ $i ] } = '0'; } --- 43,46 ---- |
|
From: C. R. D. <rya...@us...> - 2004-03-08 22:50:45
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv5486 Modified Files: excel_histogram_count.pl excel_logs_lib.pl excel_searches_count.pl Log Message: Updated scripts that produce the .count reports so that 0 counts are included, which means traversing the vocab list, so I also added a new array of vocabs to suppress (i.e. "None of the above", etc.). This means that .count reports should have a consistent number of rows from month to month, and will be easier to compare/collate. Also added "Collection=ANY" row to the histogram count reports script. Index: excel_histogram_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_histogram_count.pl,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -r1.4 -r1.5 *** excel_histogram_count.pl 13 Feb 2004 01:18:35 -0000 1.4 --- excel_histogram_count.pl 8 Mar 2004 22:26:07 -0000 1.5 *************** *** 11,16 **** # Ryan Deardorff, Dec. 2002 # Revision 1: Oct. 2003 for DDS v.3.0.x ! # Revision 2: Jan. 2003 (filter results > 10) ! # Revision 3: Feb. 2003 (get rid of keywords column) $log_in = $ARGV[ 0 ]; --- 11,19 ---- # Ryan Deardorff, Dec. 2002 # Revision 1: Oct. 2003 for DDS v.3.0.x ! # Revision 2: Jan. 2004 (filter results > 10) ! # Revision 3: Feb. 2004 (get rid of keywords column) ! # Revision 4: Mar. 2004 (add "Collection=ANY" row) ! ! require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; $log_in = $ARGV[ 0 ]; *************** *** 20,24 **** $firstLine = <IN>; while ( <IN> ) { ! if ( /^([^\t]+)\t([^\t]*).*$/ ) { if ( !$firstDate ) { $firstDate = $1; --- 23,27 ---- $firstLine = <IN>; while ( <IN> ) { ! if ( /^([^\t]+)\t([^\t]*)\t([^\t]*).*$/ ) { if ( !$firstDate ) { $firstDate = $1; *************** *** 28,31 **** --- 31,37 ---- } $criteriaCount{ $2 }++; + if ( $3 ) { + $criteriaCount{ "Collection=ANY" }++; + } $linesCount++; } *************** *** 39,46 **** $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; $outData .= "Search criteria\tTotal searches\n"; ! @kz = keys( %criteriaCount ); ! @kz = sort( @kz ); foreach $i ( 0..$#kz ) { ! $outData .= "$kz[ $i ]\t$criteriaCount{ $kz[ $i ] }\n"; } open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; --- 45,61 ---- $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; $outData .= "Search criteria\tTotal searches\n"; ! #@kz = keys( %currentVocab ); ! push( @currentVocabLabels, "Collection=ANY" ); ! @kz = @currentVocabLabels; ! @kz = sort( @currentVocabLabels ); foreach $i ( 0..$#kz ) { ! if ( ( $kz[ $i ] !~ /^Standard\=/ ) && !$noDisplayVocabs{ $kz[ $i ] } ) { ! if ( ( $kz[ $i ] !~ /^Collection\=/ ) || ( $kz[ $i ] eq "Collection=ANY" ) ) { ! if ( !$criteriaCount{ $kz[ $i ] } ) { ! $criteriaCount{ $kz[ $i ] } = '0'; ! } ! $outData .= "$kz[ $i ]\t$criteriaCount{ $kz[ $i ] }\n"; ! } ! } } open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; Index: excel_logs_lib.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_logs_lib.pl,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -r1.7 -r1.8 *** excel_logs_lib.pl 13 Feb 2004 01:18:35 -0000 1.7 --- excel_logs_lib.pl 8 Mar 2004 22:26:07 -0000 1.8 *************** *** 26,29 **** --- 26,38 ---- "ky", "Collection" ); + # Define vocabs that are for administrative use only, and don't display on the public DDS UI: + %noDisplayVocabs = ( "Grade range=DLESE:Not applicable", 1, + "Grade range=DLESE:To be supplied", 1, + "Resource type=DLESE:Offline:Physical object", 1, + "Resource type=DLESE:Text:Annotation", 1, + "Resource type=DLESE:To be supplied", 1, + "Subject=DLESE:None of the above", 1, + "Subject=DLESE:To be supplied", 1 ); + # Define the order in which criteria are presented in reports: @metaCriteriaOrder = ( "gr", "re", "su", "cs", "ky" ); *************** *** 35,40 **** $malformed = 0; # Get current vocab field=value pairs: ! &LoadCurrentVocab; # LoadCurrentVocab loads global %currentVocab with current vocab field=value pairs. --- 44,51 ---- $malformed = 0; + @currentVocabLabels = (); + # Get current vocab field=value pairs: ! &LoadCurrentVocab; # LoadCurrentVocab loads global %currentVocab with current vocab field=value pairs. *************** *** 63,66 **** --- 74,79 ---- if ( $doAnalyze{ $field } ) { $currentVocab{ "$field=$1" } = $metaCriteriaLabels{ $field } . "=$2"; + #print $metaCriteriaLabels{ $field } . "=$2\n"; + push( @currentVocabLabels, $metaCriteriaLabels{ $field } . "=$2" ); $currentVocabHist{ "$field=$1" }++; } Index: excel_searches_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_searches_count.pl,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -r1.2 -r1.3 *** excel_searches_count.pl 20 Jan 2004 19:37:25 -0000 1.2 --- excel_searches_count.pl 8 Mar 2004 22:26:08 -0000 1.3 *************** *** 11,14 **** --- 11,16 ---- # Ryan Deardorff, Dec. 2002 + require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; + $log_in = $ARGV[ 0 ]; $log_out = $ARGV[ 1 ]; *************** *** 40,43 **** --- 42,49 ---- $criteriaCount{ $criteriaOf[ $i ] }++; } + elsif ( !$criteriaCount{ $criteriaOf[ $i ] } && !$noDisplayVocabs{ $criteriaOf[ $i ] } ) { + print $criteriaOf[ $i ] . "\n"; + $criteriaCount{ $criteriaOf[ $i ] } = '0'; + } $i++; } |
|
From: Peter B. <pbu...@us...> - 2004-02-27 17:01:28
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv21361 Modified Files: monthly1 monthlyREADME Log Message: Clarified README Index: monthly1 =================================================================== RCS file: /cvsroot/dlese/logs-project/monthly1,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -r1.2 -r1.3 *** monthly1 7 Oct 2003 21:55:21 -0000 1.2 --- monthly1 27 Feb 2004 16:44:49 -0000 1.3 *************** *** 21,24 **** --- 21,25 ---- W=/www/webroot # WEBROOT LOGHOME=$W/logs + . $LOGHOME/bin/monthly_functions Index: monthlyREADME =================================================================== RCS file: /cvsroot/dlese/logs-project/monthlyREADME,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -r1.2 -r1.3 *** monthlyREADME 7 Oct 2003 19:41:02 -0000 1.2 --- monthlyREADME 27 Feb 2004 16:44:49 -0000 1.3 *************** *** 5,8 **** --- 5,10 ---- monthly1 # at 12:01 a.m. to rotate the webapp_logs # as soon as reasonably possible + + monthly1.1 # also at 12:01 a.m. to handle spread logs # monthly2 # at anytime, to remake the symbolic links and *************** *** 22,23 **** --- 24,30 ---- must not overwrite that symlink. + Logs from Aegean are gathered using Spread. For details, see: + http://swiki.dpc.ucar.edu/Project-Systems/105 + + spreadlogd is started out /etc/rc.d/rc.local. It's config is in + /usr/local/spread/etc/spreadlogd.conf |
|
From: Peter B. <pbu...@us...> - 2004-02-27 16:58:51
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv20681 Added Files: monthly1.1 Log Message: monthly1.1 also runs at 12:01 a.m. on the first, but handles the spread logs. DNS resolution is done at this time as well. Now, if only it were support to rename monthly1 to monthly1.0 in CVS... --- NEW FILE --- #!/bin/sh # # $Id: monthly1.1,v 1.1 2004/02/27 16:42:12 pburkholder Exp $ # #MONTHLY CRON at 12:01 a.m.: # cat /dev/null > local.log # foreach log # mv last month's logs to Raw/name.$LASTMONTH.log # make last months' logs immutable # hup spreadlogd to start writing to new name.log # make symlinks from name.log to name.$THISMONTH.log # # foreach log # start DNS resolution for all logs # # P. Burkholder, Feb 2004 W=/www/webroot # WEBROOT LOGHOME=$W/logs SPCONF=/usr/local/spread/etc/spreadlogd.conf SPHOME=$LOGHOME/spread . $LOGHOME/bin/monthly_functions cd $SPHOME || Die "Could not cd to $LOGHOME/webapps/dds" cat /dev/null > local.log || Warn "local not zeroed out" spread_rotate() { lastmonth_splog=`echo $splog | sed 's/.log/.'$LASTMONTH'.log/'` thismonth_splog=`echo $splog | sed 's/.log/.'$THISMONTH'.log/'` # mv last month's logs to Raw/name.$LASTMONTH.log [ "x$ISCRON" = "x" ] && echo $PROG: mv log file to Raw mv $splog Raw/$lastmonth_splog # hup spreadlogd to start writing to new name.log pkill -HUP spreadlogd [ "x$ISCRON" = "x" ] && echo $PROG: make raw file immutable sudo /usr/bin/chattr +i Raw/$lastmonth_splog || Warn "chattr +i failed" # clean up old symlnks and make symlinks from name.log to name.$THISMONTH.log [ "x$ISCRON" = "x" ] && echo $PROG: remake symlinks: if [ -L $lastmonth_splog ]; then /bin/rm $lastmonth_splog || Warn "could not rm symlink $lastmonth_splog" fi /bin/ln -s $splog $thismonth_splog || Warn "could not link for $thismonth_splog" } spread_resolve() { lastmonth_splog=`echo $splog | sed 's/.log/.'$LASTMONTH'.log/'` thismonth_splog=`echo $splog | sed 's/.log/.'$THISMONTH'.log/'` [ "x$ISCRON" = "x" ] && echo $PROG: dns resolve to $lastmonth_splog if [ -x $LOGRESOLVE ] ; then $LOGRESOLVE < Raw/$lastmonth_splog > $lastmonth_splog || Warn "logresolve of $lastmonth_splog failed" else Warn "$LOGRESOLVE not executable" fi } # find all the active logs from spreadlogd conf, parsing the conf # file for File= lines that don't have comments splogs=`/usr/bin/perl -ne 'if (($log)=m(^\s*File\s*=.*/logs/spread/(.*))i) {print $log,"\n" unless ($log =~ /local/);}'<$SPCONF` for splog in $splogs; do spread_rotate done for splog in $splogs; do spread_resolve done # To check results, find recently changed files here: # find . -mtime -1 -ls |
|
From: C. R. D. <rya...@us...> - 2004-02-23 20:29:32
|
Update of /cvsroot/dlese/logs-project/V1 In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv7646/V1 Modified Files: excel_histogram.pl excel_histogram_count.pl excel_searches_count.pl excel_searches_filter.pl Log Message: Updated the "V1" scripts (DDS v.2.x) to toss out "paged" (results > 10) searches and histogram requests Index: excel_histogram.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/V1/excel_histogram.pl,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** excel_histogram.pl 10 Sep 2003 20:37:45 -0000 1.1 --- excel_histogram.pl 23 Feb 2004 20:15:47 -0000 1.2 *************** *** 17,21 **** open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; ! print OUT "Date\tCategory Requested\tResults displayed\tRequestor's IP\n"; open( IN, $log_in ) || die "Couldn't open $log_in"; --- 17,21 ---- open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; ! print OUT "Date\tCategory Requested\tRequestor's IP\n"; open( IN, $log_in ) || die "Couldn't open $log_in"; *************** *** 52,55 **** --- 52,58 ---- $resultRangeStart = $1; $resultRangeStop = $2; + if ( $resultRangeStart > 1 ) { + $doPrint = 0; + } } else { *************** *** 64,68 **** } if ( $doPrint ) { ! print OUT "$month $day, $year $time\t$thisCriteria\t$resultRangeStart--$resultRangeStop\t$ip\n"; $histCount++; } --- 67,74 ---- } if ( $doPrint ) { ! if ( $resultRangeStart == 0 ) { ! $resultRangeStart = 1; ! } ! print OUT "$month $day, $year $time\t$thisCriteria\t$ip\n"; $histCount++; } Index: excel_histogram_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/V1/excel_histogram_count.pl,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** excel_histogram_count.pl 10 Sep 2003 20:37:45 -0000 1.1 --- excel_histogram_count.pl 23 Feb 2004 20:15:47 -0000 1.2 *************** *** 17,21 **** $firstLine = <IN>; while ( <IN> ) { ! if ( /^([^\t]+)\t([^\t]+)\t([^\t]+)\t.*$/ ) { if ( !$firstDate ) { $firstDate = $1; --- 17,21 ---- $firstLine = <IN>; while ( <IN> ) { ! if ( /^([^\t]+)\t([^\t]+)\t.*$/ ) { if ( !$firstDate ) { $firstDate = $1; *************** *** 24,28 **** $lastDate = $1; } ! $criteriaCount{ $2 . "\t" . $3 }++; $linesCount++; } --- 24,28 ---- $lastDate = $1; } ! $criteriaCount{ $2 }++; $linesCount++; } *************** *** 34,38 **** $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; ! $outData .= "Search criteria\tResults displayed\tTotal searches\n"; @kz = keys( %criteriaCount ); @kz = sort( @kz ); --- 34,38 ---- $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; ! $outData .= "Search criteria\tTotal searches\n"; @kz = keys( %criteriaCount ); @kz = sort( @kz ); Index: excel_searches_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/V1/excel_searches_count.pl,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** excel_searches_count.pl 10 Sep 2003 20:37:45 -0000 1.1 --- excel_searches_count.pl 23 Feb 2004 20:15:47 -0000 1.2 *************** *** 16,20 **** open( IN, $log_in ) || die "Couldn't open $log_in"; $firstLine = <IN>; ! if ( $firstLine =~ /^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(.*)$/ ) { $allCriteria = $1; while ( $allCriteria =~ s/^\t([^\t]+)// ) { --- 16,20 ---- open( IN, $log_in ) || die "Couldn't open $log_in"; $firstLine = <IN>; ! if ( $firstLine =~ /^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(.*)$/ ) { $allCriteria = $1; while ( $allCriteria =~ s/^\t([^\t]+)// ) { *************** *** 26,30 **** while ( <IN> ) { ! if ( /^([^\t]+)\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t([^\t]+)(\t.*)$/ ) { if ( !$firstDate ) { $firstDate = $1; --- 26,30 ---- while ( <IN> ) { ! if ( /^([^\t]+)\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(\t.*)$/ ) { if ( !$firstDate ) { $firstDate = $1; *************** *** 33,43 **** $lastDate = $1; } ! $pageRange = $2; ! $criteria = $3; $linesCount++; $i = 0; while ( $criteria =~ s/^\t([^\t]+)// ) { if ( $1 eq '1' ) { ! $criteriaCount{ $criteriaOf[ $i ] . "\t" . $pageRange }++; } $i++; --- 33,42 ---- $lastDate = $1; } ! $criteria = $2; $linesCount++; $i = 0; while ( $criteria =~ s/^\t([^\t]+)// ) { if ( $1 eq '1' ) { ! $criteriaCount{ $criteriaOf[ $i ] }++; } $i++; *************** *** 48,52 **** $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; ! $outData .= "Search criteria\tResults displayed\tTotal searches\n"; @kz = keys( %criteriaCount ); @kz = sort( @kz ); --- 47,51 ---- $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; ! $outData .= "Search criteria\tTotal searches\n"; @kz = keys( %criteriaCount ); @kz = sort( @kz ); Index: excel_searches_filter.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/V1/excel_searches_filter.pl,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** excel_searches_filter.pl 10 Sep 2003 20:37:45 -0000 1.1 --- excel_searches_filter.pl 23 Feb 2004 20:15:47 -0000 1.2 *************** *** 36,50 **** $time = &ConvertTime( $1 ); $linesCount++; ! if ( $keywords ) { ! if ( ( $lastSearch{ $ip } eq $keywords . $criteria ) && ! ( &MinutesSince( $lastTime{ $ip }, $time ) < $dup_minutes_constraint ) ) { ! print "DUPLICATE at $time against $lastTime{ $ip } = " . &MinutesSince( $lastTime{ $ip }, $time ) . "\n"; ! $dupCount++; ! } ! else { ! $outData .= $_; ! } ! $lastSearch{ $ip } = $keywords . $criteria; ! $lastTime{ $ip } = $time; } else { --- 36,41 ---- $time = &ConvertTime( $1 ); $linesCount++; ! if ( $keywords && ( $criteria =~ /\t[01]\-\-10\D/ ) ) { ! $outData .= $_; } else { *************** *** 55,58 **** --- 46,53 ---- close( IN ); + # Remove results column since all are 1--10: + $firstLine =~ s/\tResults displayed//; + $outData =~ s/\t[01]\-\-10//g; + open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; print OUT $firstLine . $outData; |
|
From: C. R. D. <rya...@us...> - 2004-02-13 01:23:54
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv28884 Modified Files: excel_histogram.pl excel_histogram_count.pl excel_logs_lib.pl excel_searches.pl excel_searches_run.pl Log Message: Fixes for the new reports format: - "Run" script now correctly runs the count script over filtered version - Removed "keywords added" column from histograms (no longer applies) - Fixed the collapsing of standards and collections into a single column (was causing innacurate data and positive values for vocabs that don't even display on the public interface) - Removed the "from collection" column in the count reports for histograms, so that each vocab criteria only appears once - Added the user session ID to the IP column, in case there is ever a need for filtering duplicate searches within a session (still unclear to me if Barb wants that or not). Even if Barb doesn't want that, having the session ID could be usefull for other analysis purposes. Index: excel_histogram.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_histogram.pl,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -r1.6 -r1.7 *** excel_histogram.pl 20 Jan 2004 19:37:25 -0000 1.6 --- excel_histogram.pl 13 Feb 2004 01:18:35 -0000 1.7 *************** *** 11,14 **** --- 11,15 ---- # Revision 1: Oct. 2003 for DDS v.3.0.x # Revision 2: Jan. 2004 (filter results > 10) + # Revision 3: Feb. 2003 (get rid of keywords column) require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; *************** *** 19,23 **** open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; ! print OUT "Date\tCategory Requested\tFrom collection\tKeyword(s) added\tRequestor's IP\n"; open( IN, $log_in ) || die "Couldn't open $log_in"; --- 20,24 ---- open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; ! print OUT "Date\tCategory Requested\tFrom collection\tRequestor's IP\n"; open( IN, $log_in ) || die "Couldn't open $log_in"; *************** *** 57,64 **** } $resultRangeStop = $resultRangeStart + 9; ! $collectionLabel = $metaMeaning{ 'ky=' . $thisCollection }; $collectionLabel =~ s/^Collection=//; if ( $resultRangeStart == 1 ) { ! print OUT "$month $day, $year $time\t$metaMeaning{ $thisCriteria }\t$collectionLabel\t$thisKeywords\t$ip\n"; } $histCount++; --- 58,65 ---- } $resultRangeStop = $resultRangeStart + 9; ! $collectionLabel = $currentVocab{ 'ky=' . $thisCollection }; $collectionLabel =~ s/^Collection=//; if ( $resultRangeStart == 1 ) { ! print OUT "$month $day, $year $time\t$currentVocab{ $thisCriteria }\t$collectionLabel\t$ip\n"; } $histCount++; Index: excel_histogram_count.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_histogram_count.pl,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -r1.3 -r1.4 *** excel_histogram_count.pl 20 Jan 2004 19:37:25 -0000 1.3 --- excel_histogram_count.pl 13 Feb 2004 01:18:35 -0000 1.4 *************** *** 12,15 **** --- 12,16 ---- # Revision 1: Oct. 2003 for DDS v.3.0.x # Revision 2: Jan. 2003 (filter results > 10) + # Revision 3: Feb. 2003 (get rid of keywords column) $log_in = $ARGV[ 0 ]; *************** *** 19,23 **** $firstLine = <IN>; while ( <IN> ) { ! if ( /^([^\t]+)\t([^\t]*)\t([^\t]*)\t([^\t]*).*$/ ) { if ( !$firstDate ) { $firstDate = $1; --- 20,24 ---- $firstLine = <IN>; while ( <IN> ) { ! if ( /^([^\t]+)\t([^\t]*).*$/ ) { if ( !$firstDate ) { $firstDate = $1; *************** *** 26,30 **** $lastDate = $1; } ! $criteriaCount{ $2 . "\t" . $3 . "\t" . $4 }++; $linesCount++; } --- 27,31 ---- $lastDate = $1; } ! $criteriaCount{ $2 }++; $linesCount++; } *************** *** 37,41 **** $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; ! $outData .= "Search criteria\tFrom collection\tKeyword(s) added\tTotal searches\n"; @kz = keys( %criteriaCount ); @kz = sort( @kz ); --- 38,42 ---- $outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n"; ! $outData .= "Search criteria\tTotal searches\n"; @kz = keys( %criteriaCount ); @kz = sort( @kz ); Index: excel_logs_lib.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_logs_lib.pl,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -r1.6 -r1.7 *** excel_logs_lib.pl 20 Jan 2004 19:37:25 -0000 1.6 --- excel_logs_lib.pl 13 Feb 2004 01:18:35 -0000 1.7 *************** *** 64,68 **** $currentVocab{ "$field=$1" } = $metaCriteriaLabels{ $field } . "=$2"; $currentVocabHist{ "$field=$1" }++; - $metaMeaning{ "$field=$1" } = $metaCriteriaLabels{ $field } . "=" . $2; } } --- 64,67 ---- Index: excel_searches.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_searches.pl,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -r1.6 -r1.7 *** excel_searches.pl 20 Jan 2004 19:37:25 -0000 1.6 --- excel_searches.pl 13 Feb 2004 01:18:35 -0000 1.7 *************** *** 6,12 **** # Format for use: # ! # excel_searches_vocabs_collapsed.pl [input_log] [output_file] # ! # Ryan Deardorff, Jan. 2004 for DDS v3.0.x require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; --- 6,16 ---- # Format for use: # ! # excel_searches.pl [input_log] [output_file] [filter_vocabs?] # ! # The last argument can be anything, and its existence specifies that vocab ! # filtering should be applied (collapse standards and collections into 1 ! # column). ! # ! # Ryan Deardorff, Feb. 2004 for DDS v3.x require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl"; *************** *** 24,28 **** open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; ! print OUT "Date\tRequestor's IP\t# Search results\t# Resources in library\tSearch value\tResults displayed"; @key = keys( %currentVocab ); @key = sort( @key ); --- 28,32 ---- open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out"; ! print OUT "Date\tRequestor's IP (Session ID)\t# Search results\t# Resources in library\tSearch value\tResults displayed"; @key = keys( %currentVocab ); @key = sort( @key ); *************** *** 30,41 **** foreach $j ( 0..$#key ) { $currentVocab{ $key[ $j ] } =~ /^(.*)=(.*)/; ! if ( $FILTER_VOCABS{ $1 } ) { # Collapsed view of a vocab (i.e. Standards) ! if ( !$renderedVocab{ $1 } ) { ! print OUT "\t$1=ANY"; ! $renderedVocab{ $1 }++; } ! } ! else { ! if ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) { print OUT "\t$currentVocab{ $key[ $j ] }"; } --- 34,46 ---- foreach $j ( 0..$#key ) { $currentVocab{ $key[ $j ] } =~ /^(.*)=(.*)/; ! $theKey = $1; ! if ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) { ! if ( $FILTER_VOCABS{ $theKey } ) { # Collapsed view of a vocab (i.e. Standards) ! if ( !$renderedVocab{ $theKey } ) { ! print OUT "\t$theKey=ANY"; ! $renderedVocab{ $theKey }++; ! } } ! else { print OUT "\t$currentVocab{ $key[ $j ] }"; } *************** *** 56,60 **** $year = $4; $time = $5; ! if ( !&IsFilterEntry( $logEntry ) && ( /q\=\S+/ ) && !( /\/query\.do\?hist\=true/ ) ) { if ( /dlese\.org\s+80\s+search\s+(\d+)\s+(\d+)\s+(\-?\d+)[^\"]\".*\"<qle>(.*)<\/qle>\"/ ) { $numResults = $1; --- 61,65 ---- $year = $4; $time = $5; ! if ( !&IsFilterEntry( $logEntry ) && ( /q\=\S+/ ) && !( /hist\=true/ ) ) { if ( /dlese\.org\s+80\s+search\s+(\d+)\s+(\d+)\s+(\-?\d+)[^\"]\".*\"<qle>(.*)<\/qle>\"/ ) { $numResults = $1; *************** *** 67,70 **** --- 72,76 ---- $resultRangeStart++; } + $sessionId = ''; # NOTE: If we ever provide an option for # of results per page, the following will # need to be done differently: *************** *** 88,91 **** --- 94,100 ---- } } + elsif ( $1 eq 'session' ) { + $sessionId = " ($2)"; + } } #die; *************** *** 96,118 **** $keywordTerms =~ s/\\([\"\'])/$1/g; # The new system escapes quotes, so unescape them $keywordTerms =~ s/\"/\"\"/g; # Turn " into "" (so they show in Excel and aren't used as delimeters) ! print OUT "$month $day, $year $time\t$ip\t$numResults\t$totalResources\t\"$keywordTerms\"\t$resultRangeStart--$resultRangeStop"; foreach $i ( 0..$#metaCriteriaOrder ) { foreach $j ( 0..$#key ) { $currentVocab{ $key[ $j ] } =~ /^(.*)=(.*)/; ! if ( $FILTER_VOCABS{ $1 } ) { # Collapsed view of a vocab ! if ( !$renderFilteredVocab{ $1 } ) { ! print OUT "\t1"; ! $renderFilteredVocab{ $1 }++; ! } ! } ! elsif ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) { ! if ( $theseCriteria{ $key[ $j ] } ) { ! print OUT "\t1"; } else { ! print OUT "\t0"; } } } } print OUT "\n"; --- 105,142 ---- $keywordTerms =~ s/\\([\"\'])/$1/g; # The new system escapes quotes, so unescape them $keywordTerms =~ s/\"/\"\"/g; # Turn " into "" (so they show in Excel and aren't used as delimeters) ! print OUT "$month $day, $year $time\t${ip}${sessionId}\t$numResults\t$totalResources\t\"$keywordTerms\"\t$resultRangeStart--$resultRangeStop"; ! $filteredCacheValue = ''; foreach $i ( 0..$#metaCriteriaOrder ) { + $filteredCacheValue = ''; foreach $j ( 0..$#key ) { $currentVocab{ $key[ $j ] } =~ /^(.*)=(.*)/; ! $theKey = $1; ! if ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) { ! if ( $FILTER_VOCABS{ $theKey } ) { # Collapsed view of a vocab ! if ( $filteredCacheValue eq '' ) { ! $filteredCacheValue = "\t0"; ! } ! if ( $theseCriteria{ $key[ $j ] } ) { ! $filteredCacheValue = "\t1"; ! } } else { ! if ( $filteredCacheValue ne '' ) { ! print OUT $filteredCacheValue; ! $filteredCacheValue = ''; ! } ! if ( $theseCriteria{ $key[ $j ] } ) { ! print OUT "\t1"; ! } ! else { ! print OUT "\t0"; ! } } } } + if ( $filteredCacheValue ne '' ) { + print OUT $filteredCacheValue; + $filteredCacheValue = ''; + } } print OUT "\n"; Index: excel_searches_run.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_searches_run.pl,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -r1.2 -r1.3 *** excel_searches_run.pl 20 Jan 2004 19:37:25 -0000 1.2 --- excel_searches_run.pl 13 Feb 2004 01:18:35 -0000 1.3 *************** *** 21,25 **** &systemPrint( "excel_searches.pl $log ${output}.xls -c" ); &systemPrint( "excel_searches_filter.pl ${output}.xls ${output}.filtered.xls" ); ! &systemPrint( "excel_searches_count.pl ${output}.xls ${output}.count.xls" ); sub systemPrint { --- 21,25 ---- &systemPrint( "excel_searches.pl $log ${output}.xls -c" ); &systemPrint( "excel_searches_filter.pl ${output}.xls ${output}.filtered.xls" ); ! &systemPrint( "excel_searches_count.pl ${output}.filtered.xls ${output}.count.xls" ); sub systemPrint { |
|
From: C. R. D. <rya...@us...> - 2004-01-20 19:46:27
|
Update of /cvsroot/dlese/logs-project
In directory sc8-pr-cvs1:/tmp/cvs-serv16283
Modified Files:
CHANGES.txt
Log Message:
More doc tweaks
Index: CHANGES.txt
===================================================================
RCS file: /cvsroot/dlese/logs-project/CHANGES.txt,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -r1.2 -r1.3
*** CHANGES.txt 20 Jan 2004 19:37:25 -0000 1.2
--- CHANGES.txt 20 Jan 2004 19:46:24 -0000 1.3
***************
*** 28,33 ****
1) Excel search reports:
- - Current reports still generated with all vocabs specified, but are renamed only for
- use by count reports (much easier to see all vocabs as rows), then deleted
- New "collapsed" search and browse reports take place of previous reports:
- Certain vocabs ("Standard", "Collection" for DDS v3.0.x) indicated in 1 column
--- 28,31 ----
|
|
From: C. R. D. <rya...@us...> - 2004-01-20 19:44:14
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1:/tmp/cvs-serv15750 Modified Files: excel_README.txt Log Message: Doc tweak Index: excel_README.txt =================================================================== RCS file: /cvsroot/dlese/logs-project/excel_README.txt,v retrieving revision 1.1.1.1 retrieving revision 1.2 diff -C2 -r1.1.1.1 -r1.2 *** excel_README.txt 6 Feb 2003 19:19:02 -0000 1.1.1.1 --- excel_README.txt 20 Jan 2004 19:44:11 -0000 1.2 *************** *** 30,33 **** --- 30,34 ---- Subset of the first spreadsheet, where the following have been filtered out: -Empty searches + -"Paged" searches (results > 10) [output_log].count.xls |
|
From: C. R. D. <rya...@us...> - 2004-01-20 19:37:28
|
Update of /cvsroot/dlese/logs-project
In directory sc8-pr-cvs1:/tmp/cvs-serv13530
Modified Files:
CHANGES.txt excel_histogram.pl excel_histogram_count.pl
excel_histogram_run.pl excel_logs_lib.pl excel_searches.pl
excel_searches_count.pl excel_searches_filter.pl
excel_searches_run.pl
Log Message:
Changes requested by Barb in the search and browse excel sheets:
- "Paged" searches (results > 10) thrown out (except in main search report)
- Added ability to specify certain vocabs that get collapsed into a single
column/row, and configured to do so with Collection and Standards
- These changes ONLY done to the V2 scripts so far
Index: CHANGES.txt
===================================================================
RCS file: /cvsroot/dlese/logs-project/CHANGES.txt,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -r1.1 -r1.2
*** CHANGES.txt 2 Oct 2003 22:52:50 -0000 1.1
--- CHANGES.txt 20 Jan 2004 19:37:25 -0000 1.2
***************
*** 24,25 ****
--- 24,37 ----
- Report labels are now the fully-qualified metadata names, i.e. "DLESE:Space science"
+ _________________________________________________________________________________________
+ Revision 2 (DDS v3.0.x) Jan. 2004
+
+ 1) Excel search reports:
+ - Current reports still generated with all vocabs specified, but are renamed only for
+ use by count reports (much easier to see all vocabs as rows), then deleted
+ - New "collapsed" search and browse reports take place of previous reports:
+ - Certain vocabs ("Standard", "Collection" for DDS v3.0.x) indicated in 1 column
+ (ANY selected)
+
+ 2) Excel filtered and count reports:
+ - All "paged" searches (results > 10) are now thrown out
Index: excel_histogram.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_histogram.pl,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -r1.5 -r1.6
*** excel_histogram.pl 3 Oct 2003 20:41:35 -0000 1.5
--- excel_histogram.pl 20 Jan 2004 19:37:25 -0000 1.6
***************
*** 10,13 ****
--- 10,14 ----
# Ryan Deardorff, Dec. 2002
# Revision 1: Oct. 2003 for DDS v.3.0.x
+ # Revision 2: Jan. 2004 (filter results > 10)
require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl";
***************
*** 18,22 ****
open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out";
! print OUT "Date\tCategory Requested\tFrom collection\tKeyword(s) added\tResults displayed\tRequestor's IP\n";
open( IN, $log_in ) || die "Couldn't open $log_in";
--- 19,23 ----
open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out";
! print OUT "Date\tCategory Requested\tFrom collection\tKeyword(s) added\tRequestor's IP\n";
open( IN, $log_in ) || die "Couldn't open $log_in";
***************
*** 58,62 ****
$collectionLabel = $metaMeaning{ 'ky=' . $thisCollection };
$collectionLabel =~ s/^Collection=//;
! print OUT "$month $day, $year $time\t$metaMeaning{ $thisCriteria }\t$collectionLabel\t$thisKeywords\t$resultRangeStart--$resultRangeStop\t$ip\n";
$histCount++;
}
--- 59,65 ----
$collectionLabel = $metaMeaning{ 'ky=' . $thisCollection };
$collectionLabel =~ s/^Collection=//;
! if ( $resultRangeStart == 1 ) {
! print OUT "$month $day, $year $time\t$metaMeaning{ $thisCriteria }\t$collectionLabel\t$thisKeywords\t$ip\n";
! }
$histCount++;
}
Index: excel_histogram_count.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_histogram_count.pl,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -r1.2 -r1.3
*** excel_histogram_count.pl 2 Oct 2003 22:52:50 -0000 1.2
--- excel_histogram_count.pl 20 Jan 2004 19:37:25 -0000 1.3
***************
*** 11,14 ****
--- 11,15 ----
# Ryan Deardorff, Dec. 2002
# Revision 1: Oct. 2003 for DDS v.3.0.x
+ # Revision 2: Jan. 2003 (filter results > 10)
$log_in = $ARGV[ 0 ];
***************
*** 18,22 ****
$firstLine = <IN>;
while ( <IN> ) {
! if ( /^([^\t]+)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t.*$/ ) {
if ( !$firstDate ) {
$firstDate = $1;
--- 19,23 ----
$firstLine = <IN>;
while ( <IN> ) {
! if ( /^([^\t]+)\t([^\t]*)\t([^\t]*)\t([^\t]*).*$/ ) {
if ( !$firstDate ) {
$firstDate = $1;
***************
*** 25,29 ****
$lastDate = $1;
}
! $criteriaCount{ $2 . "\t" . $3 . "\t" . $4 . "\t" . $5 }++;
$linesCount++;
}
--- 26,30 ----
$lastDate = $1;
}
! $criteriaCount{ $2 . "\t" . $3 . "\t" . $4 }++;
$linesCount++;
}
***************
*** 36,40 ****
$outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n";
! $outData .= "Search criteria\tFrom collection\tKeyword(s) added\tResults displayed\tTotal searches\n";
@kz = keys( %criteriaCount );
@kz = sort( @kz );
--- 37,41 ----
$outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n";
! $outData .= "Search criteria\tFrom collection\tKeyword(s) added\tTotal searches\n";
@kz = keys( %criteriaCount );
@kz = sort( @kz );
Index: excel_histogram_run.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_histogram_run.pl,v
retrieving revision 1.1.1.1
retrieving revision 1.2
diff -C2 -r1.1.1.1 -r1.2
*** excel_histogram_run.pl 6 Feb 2003 19:19:02 -0000 1.1.1.1
--- excel_histogram_run.pl 20 Jan 2004 19:37:25 -0000 1.2
***************
*** 17,24 ****
$output = $ARGV[ 1 ];
! print "Running excel_histogram.pl...\n";
! system( "excel_histogram.pl $log ${output}.xls" );
! print "Running excel_histogram_count.pl...\n";
! system( "excel_histogram_count.pl ${output}.xls ${output}.count.xls" );
exit( 0 );
--- 17,27 ----
$output = $ARGV[ 1 ];
! &systemPrint( "excel_histogram.pl $log ${output}.xls" );
! &systemPrint( "excel_histogram_count.pl ${output}.xls ${output}.count.xls" );
!
! sub systemPrint {
! print "Running script: " . $_[ 0 ] . "\n";
! system( $_[ 0 ] );
! }
exit( 0 );
Index: excel_logs_lib.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_logs_lib.pl,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -r1.5 -r1.6
*** excel_logs_lib.pl 2 Oct 2003 22:52:50 -0000 1.5
--- excel_logs_lib.pl 20 Jan 2004 19:37:25 -0000 1.6
***************
*** 81,93 ****
print "\t" . $kz[ $i ] . "\n";
}
! print <<EOF;
!
! Please create a file called 'criteria.key' in the same directory where you run the
! scripts from that maps old field=value pairs to new ones, and then re-run the scripts.
! Each line of this key file should be in the following format:
!
! oldField=oldValue[TAB]newField=newValue
!
! EOF
}
}
--- 81,94 ----
print "\t" . $kz[ $i ] . "\n";
}
! # The following no longer needed, since vocab values can't change (so an error really means an error!):
! # print <<EOF;
! #
! #Please create a file called 'criteria.key' in the same directory where you run the
! #scripts from that maps old field=value pairs to new ones, and then re-run the scripts.
! #Each line of this key file should be in the following format:
! #
! #oldField=oldValue[TAB]newField=newValue
! #
! #EOF
}
}
Index: excel_searches.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_searches.pl,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -r1.5 -r1.6
*** excel_searches.pl 3 Oct 2003 20:41:35 -0000 1.5
--- excel_searches.pl 20 Jan 2004 19:37:25 -0000 1.6
***************
*** 6,17 ****
# Format for use:
#
! # excel_searches.pl [input_log] [output_file]
#
! # Ryan Deardorff, Dec. 2002
! # Revision 1: Oct. 2003 for DDS v.3.0.x
require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl";
require "$ENV{ LOG_SCRIPTS }/filters.pl"; # NCSA filtering
$log_in = $ARGV[ 0 ];
$log_out = $ARGV[ 1 ];
--- 6,23 ----
# Format for use:
#
! # excel_searches_vocabs_collapsed.pl [input_log] [output_file]
#
! # Ryan Deardorff, Jan. 2004 for DDS v3.0.x
require "$ENV{ LOG_SCRIPTS }/excel_logs_lib.pl";
require "$ENV{ LOG_SCRIPTS }/filters.pl"; # NCSA filtering
+ # The following will collapse vocab columns down to one, indicating that ANY selection was made in the given category:
+ %FILTER_VOCABS = ();
+ if ( $ARGV[ 2 ] ) {
+ %FILTER_VOCABS = ( 'Standard', 1,
+ 'Collection', 1 );
+ }
+
$log_in = $ARGV[ 0 ];
$log_out = $ARGV[ 1 ];
***************
*** 23,28 ****
foreach $i ( 0..$#metaCriteriaOrder ) {
foreach $j ( 0..$#key ) {
! if ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) {
! print OUT "\t$currentVocab{ $key[ $j ] }";
}
}
--- 29,43 ----
foreach $i ( 0..$#metaCriteriaOrder ) {
foreach $j ( 0..$#key ) {
! $currentVocab{ $key[ $j ] } =~ /^(.*)=(.*)/;
! if ( $FILTER_VOCABS{ $1 } ) { # Collapsed view of a vocab (i.e. Standards)
! if ( !$renderedVocab{ $1 } ) {
! print OUT "\t$1=ANY";
! $renderedVocab{ $1 }++;
! }
! }
! else {
! if ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) {
! print OUT "\t$currentVocab{ $key[ $j ] }";
! }
}
}
***************
*** 84,88 ****
foreach $i ( 0..$#metaCriteriaOrder ) {
foreach $j ( 0..$#key ) {
! if ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) {
if ( $theseCriteria{ $key[ $j ] } ) {
print OUT "\t1";
--- 99,110 ----
foreach $i ( 0..$#metaCriteriaOrder ) {
foreach $j ( 0..$#key ) {
! $currentVocab{ $key[ $j ] } =~ /^(.*)=(.*)/;
! if ( $FILTER_VOCABS{ $1 } ) { # Collapsed view of a vocab
! if ( !$renderFilteredVocab{ $1 } ) {
! print OUT "\t1";
! $renderFilteredVocab{ $1 }++;
! }
! }
! elsif ( $key[ $j ] =~ /^$metaCriteriaOrder[ $i ]\=/ ) {
if ( $theseCriteria{ $key[ $j ] } ) {
print OUT "\t1";
Index: excel_searches_count.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_searches_count.pl,v
retrieving revision 1.1.1.1
retrieving revision 1.2
diff -C2 -r1.1.1.1 -r1.2
*** excel_searches_count.pl 6 Feb 2003 19:19:02 -0000 1.1.1.1
--- excel_searches_count.pl 20 Jan 2004 19:37:25 -0000 1.2
***************
*** 26,30 ****
while ( <IN> ) {
! if ( /^([^\t]+)\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t([^\t]+)(\t.*)$/ ) {
if ( !$firstDate ) {
$firstDate = $1;
--- 26,30 ----
while ( <IN> ) {
! if ( /^([^\t]+)\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+(\t.*)$/ ) {
if ( !$firstDate ) {
$firstDate = $1;
***************
*** 33,43 ****
$lastDate = $1;
}
! $pageRange = $2;
! $criteria = $3;
$linesCount++;
$i = 0;
while ( $criteria =~ s/^\t([^\t]+)// ) {
if ( $1 eq '1' ) {
! $criteriaCount{ $criteriaOf[ $i ] . "\t" . $pageRange }++;
}
$i++;
--- 33,42 ----
$lastDate = $1;
}
! $criteria = $2;
$linesCount++;
$i = 0;
while ( $criteria =~ s/^\t([^\t]+)// ) {
if ( $1 eq '1' ) {
! $criteriaCount{ $criteriaOf[ $i ] }++;
}
$i++;
***************
*** 48,52 ****
$outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n";
! $outData .= "Search criteria\tResults displayed\tTotal searches\n";
@kz = keys( %criteriaCount );
@kz = sort( @kz );
--- 47,51 ----
$outData = "$linesCount searches conducted from $firstDate - $lastDate\n\n";
! $outData .= "Search criteria\tTotal searches\n";
@kz = keys( %criteriaCount );
@kz = sort( @kz );
Index: excel_searches_filter.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_searches_filter.pl,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -r1.2 -r1.3
*** excel_searches_filter.pl 12 Feb 2003 23:39:44 -0000 1.2
--- excel_searches_filter.pl 20 Jan 2004 19:37:25 -0000 1.3
***************
*** 5,50 ****
#
# 1) Entries where the search field is blank
! # 2) Duplicate searches (back-to-back) conducted by the same IP within the given timeperiod
#
# Format for use:
#
! # excel_filter.pl [input_log] [output_file] [minutes_constraint]
! #
! # minutes_constraint specifies the number of minutes within which to detect and filter
! # duplicate searches. This is optional, and if not specified, a default of 60 min. will
! # be used.
#
# Ryan Deardorff, Dec. 2002
!
! require "$ENV{ LOG_SCRIPTS }/time.pl";
$log_in = $ARGV[ 0 ];
$log_out = $ARGV[ 1 ];
- $dup_minutes_constraint = $ARGV[ 2 ];
- if ( $dup_minutes_constraint eq '' ) {
- $dup_minutes_constraint = 60;
- }
-
open( IN, $log_in ) || die "Couldn't open $log_in";
$firstLine = <IN>;
while ( <IN> ) {
! if ( /^([^\t]+)\t([^\t]+)\t[^\t]+\t[^\t]+\t\"([^\t]*)\"(\t.*)$/ ) {
! $ip = $2;
! $keywords = $3;
! $criteria = $4;
! $time = &ConvertTime( $1 );
$linesCount++;
! if ( $keywords ) {
! if ( ( $lastSearch{ $ip } eq $keywords . $criteria ) &&
! ( &MinutesSince( $lastTime{ $ip }, $time ) < $dup_minutes_constraint ) ) {
! print "DUPLICATE at $time against $lastTime{ $ip } = " . &MinutesSince( $lastTime{ $ip }, $time ) . "\n";
! $dupCount++;
! }
! else {
! $outData .= $_;
! }
! $lastSearch{ $ip } = $keywords . $criteria;
! $lastTime{ $ip } = $time;
}
else {
--- 5,29 ----
#
# 1) Entries where the search field is blank
! # 2) "Paged" searches, or search results > 10
#
# Format for use:
#
! # excel_filter.pl [input_log] [output_file]
#
# Ryan Deardorff, Dec. 2002
! # Last updated Jan. 2004 (remove dup check and filter only results > 10)
$log_in = $ARGV[ 0 ];
$log_out = $ARGV[ 1 ];
open( IN, $log_in ) || die "Couldn't open $log_in";
$firstLine = <IN>;
while ( <IN> ) {
! if ( /^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+\t\"([^\t]*)\"(\t.*)$/ ) {
! $keywords = $1;
! $page = $2;
$linesCount++;
! if ( $keywords && ( $page =~ /^\s*\d\-\-10\D/ ) ) {
! $outData .= $_;
}
else {
***************
*** 55,75 ****
close( IN );
open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out";
print OUT $firstLine . $outData;
close( OUT );
! if ( !$dupCount ) { $dupCount = '0'; }
!
! print "Processed $linesCount entries, and threw out $emptyCount entries with empty keywords, and $dupCount duplicate (back-to-back) entries.\n";
exit( 0 );
- sub ConvertTime {
- my( $ret ) = '';
- my( $t ) = $_[ 0 ];
- if ( $t =~ /^(\S+)\s+(\d+)\,\s+(\d+)\s+(\S+)/ ) {
- $ret = $2 . "/" . $MonthVal{ $1 } . "/" . $3 . " " . $4;
- }
- else { die "Unrecognized time format: $t\n"; }
- return $ret;
- }
--- 34,47 ----
close( IN );
+ # Remove results column since all are 1--10:
+ $firstLine =~ s/\tResults displayed//;
+ $outData =~ s/\t1\-\-10//g;
+
open( OUT, ">" . $log_out ) || die "Couldn't write to $log_out";
print OUT $firstLine . $outData;
close( OUT );
! print "Processed $linesCount entries, and threw out $emptyCount entries with empty keywords (or results > 10), and $dupCount duplicate (back-to-back) entries.\n";
exit( 0 );
Index: excel_searches_run.pl
===================================================================
RCS file: /cvsroot/dlese/logs-project/excel_searches_run.pl,v
retrieving revision 1.1.1.1
retrieving revision 1.2
diff -C2 -r1.1.1.1 -r1.2
*** excel_searches_run.pl 6 Feb 2003 19:19:02 -0000 1.1.1.1
--- excel_searches_run.pl 20 Jan 2004 19:37:25 -0000 1.2
***************
*** 14,27 ****
#
# Ryan Deardorff, Dec. 2002
$log = $ARGV[ 0 ];
$output = $ARGV[ 1 ];
! print "Running excel_searches.pl...\n";
! system( "excel_searches.pl $log ${output}.xls" );
! print "Running excel_searches_filter.pl...\n";
! system( "excel_searches_filter.pl ${output}.xls ${output}.filtered.xls 0" );
! print "Running excel_searches_count.pl...\n";
! system( "excel_searches_count.pl ${output}.filtered.xls ${output}.count.xls" );
exit( 0 );
--- 14,30 ----
#
# Ryan Deardorff, Dec. 2002
+ # Last updated Jan. 2004 (vocab and paged filtering)
$log = $ARGV[ 0 ];
$output = $ARGV[ 1 ];
! &systemPrint( "excel_searches.pl $log ${output}.xls -c" );
! &systemPrint( "excel_searches_filter.pl ${output}.xls ${output}.filtered.xls" );
! &systemPrint( "excel_searches_count.pl ${output}.xls ${output}.count.xls" );
!
! sub systemPrint {
! print "Running script: " . $_[ 0 ] . "\n";
! system( $_[ 0 ] );
! }
exit( 0 );
|
|
From: Peter B. <pbu...@us...> - 2003-11-20 17:51:41
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1:/tmp/cvs-serv24933 Modified Files: monthly2 Log Message: monthly2 hosts lists changed to accomodate preview and beta-geo hosts -peterb Index: monthly2 =================================================================== RCS file: /cvsroot/dlese/logs-project/monthly2,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** monthly2 7 Oct 2003 19:37:15 -0000 1.1 --- monthly2 20 Nov 2003 17:51:28 -0000 1.2 *************** *** 13,17 **** # P. Burkholder, March 2003, September 2003 ! HOSTS="beta/beta catalog/catalog dpc/dpc dwel/dwel flood/flood nsdl/nsdl oai/oai page/page" HOSTS="$HOSTS rd/rd swiki/swiki_dpc swiki/swiki_dlese training/training v1/v1_dlese v1/v1_dpc" TESTHOSTS="bogus/swiki_dpc bogus/swiki_dlese" --- 13,18 ---- # P. Burkholder, March 2003, September 2003 ! HOSTS="beta/beta beta/beta-geo beta/preview catalog/catalog" ! HOSTS="$HOSTS dpc/dpc dwel/dwel flood/flood nsdl/nsdl oai/oai page/page" HOSTS="$HOSTS rd/rd swiki/swiki_dpc swiki/swiki_dlese training/training v1/v1_dlese v1/v1_dpc" TESTHOSTS="bogus/swiki_dpc bogus/swiki_dlese" |
|
From: C. R. D. <rya...@us...> - 2003-10-07 23:11:52
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1:/tmp/cvs-serv2588 Modified Files: merge_logs.pl run_log_scripts.pl Added Files: docs-logs.html Log Message: Tweaked merge script to handle reading .gz files through a zcat pipe, and added the documentation HTML. --- NEW FILE --- <html> <head> <title> Documentation for DLESE Log Processing and Analysis </title> </head> <body> <h1> Documentation for DLESE Log Processing and Analysis </h1> <p><a href="/projects/libdev/docs/index.html">Return to DLESE V2 index</a> </p> <h2> Version </h2> <p>2.0</p> <h2> Description </h2> <!-- Description and purpose of the component. Include links to diagrams as appropriate. --> <p>When users of the DLESE library visit the site, they leave a text footprint behind. In addition to the standard NCSA traffic logs, which reflect the URL of every page visited and some other key items like response code and number of bytes transfered, DLESE is keeping a seperate set of logs that include DLESE Discovery System (DDS) specific information, such as how many resources were in the library at the time of a given search, and how many search results the user was presented with in a given search.</p> <h2> Audience </h2> <ul> <li>Evaluators of the library</li> <li>UI designers interested in the use patterns found within the analysis reports</li> </ul> <h2> How this component is used </h2> <p>Evaluators of the library use the information found within the logs to track the evolution of site visitation, to see what users of the site are looking for, how successful they were in finding it, and to therefore guide future development of the library. Similarly, user interface designers can find information that helps them develop the next version of the UI.</p> <h2> Functional requirements </h2> <ul> <li>Gather as much information about library use as possible to help evaluation, including: <ul> <li>Standard NCSA traffic logs for internal and external partner sites</li> <li><a href="http://swiki.dpc.ucar.edu/Project-Discovery/39">URL Redirects</a> that indicate which resources are being visited through the Discovery system</li> <li>User search results: <ul> <li>Keyword</li> <li>Selected criteria (vocabularies)</li> <li>Number of resources returned</li> <li>Total number of resources in the library</li> <li>First result index (starting at 0), or results page times 10</li> </ul> </li> </ul> </li> <li>Process the information collected on a month by month basis: <ul> <li>Merge all site/partner NCSA logs for unified processing by Webtrends</li> <li>Produce various Excel reports: <ul> <li>Redirect link (resource visit) tracking</li> <li>Site search results by "type": <ul> <li>search</li> <li>histogram (View all resources and View collections)</li> </ul> </li> </ul> </li> </ul> </li> </ul> <h2> Implementation technologies and rationale </h2> <!-- What technologies and why: --> <ul> <li>Gathering of information is done through standard Apache NCSA logging mechanisms, as well as custom logging through the Java back-end of the DDS.</li> <li>Perl scripts for log merging and excel report creation, as Perl is unmatched in ease of text parsing using regular expressions.</li> </ul> <h2> Development process and workflow </h2> <!-- Description of the development process and a typical workflow. Include tools used such as Ant, CVS, JEdit, Dreamweaver, Spy, scripts, etc. Include specifics such as what machines things are run on, what directories data is saved to, etc. such that an outsider might be able to participate in the development process after reading this description. --> <p></p> <h2> Testing and maintenance procedures </h2> <!-- Describe the procedures used to test the component for accuracy, performance and bugs and how it is updated and maintained over time. Include testing tools such as spell checkers, JUnit, JMeter, debuggers, etc. --> <p>When data is collected from the NCSA logs for processing (merge) or report creation, the output is compared against what is expected by examining the original logs with tools such as 'grep' and 'wc'. Calculators, text editors, and Excel are also used to examine both the input and output text (or .xls) files.</p> <h2> Library services provided </h2> <!-- Does this component act as a service to other components, either internal or external? If so, describe. --> <p>Currently, there are none. The information collected could be used in any application that needs it, probably most easily through SQL (see <a href='#future' title='Future development'>Future development</a>). </p> <h2> Other components used by this component </h2> <!-- List of other components that this component uses and/or interacts with, both internal and external to the DPC. Include links to documentation and descriptions of those components if possible --> <ul> <li><a href="docs-vocab-manager.html">Vocabularies Manager</a> is used to produce search criteria headers in the Excel reports (since they are now logged as, e.g. "cs=004")</li> </ul> <a name='future'></a> <h2> Future development </h2> <!-- List additional features and functionality that is anticipated for future development of the component --> <ul> <li>Script that processes the logs and populates MySQL tables, so that "count" type reports, etc., can be generated more easily, and more quickly. phpMyAdmin can also then be used for playing with queries and examining the data all sorts of ways.</li> <li>Use the information collected to generate parts of the site, for example: <ul> <li>List of the most visited sites</li> <li>Boost resources in searches by frequency of visit</li> <li>"Users who searched for your keywords also searched for..."</li> </ul> </li> <li>Guide the building of the library through examination of most frequent search terms, most frequently visited sites, and controlled vocabulary use.</li> </ul> <h2> Links to additional documentation </h2> <!-- Links to JavaDocs, diagrams or additional documentation relevant to this component --> <ul> <li><a href="http://swiki.dpc.ucar.edu/Project-Discovery/39">Resource visit tracking - redirect logging syntax</a></li> <li><a href="http://swiki.dpc.ucar.edu/Project-Discovery/27">Query logging and assessments</a></li> <li><a href="http://swiki.dpc.ucar.edu/Project-Discovery/24">Query logging considerations</a></li> </ul> <p> <i>Contact: <a href="mailto:su...@dl...">su...@dl...</a></i><br> <i>Last updated: October 6, 2003</i><br> <i>Author: Ryan Deardorff</i><br> </p> </body> </html> Index: merge_logs.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/merge_logs.pl,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -r1.7 -r1.8 *** merge_logs.pl 7 Apr 2003 19:56:57 -0000 1.7 --- merge_logs.pl 7 Oct 2003 23:11:44 -0000 1.8 *************** *** 106,111 **** $datestamp = $1; } ! open( LOG1, $_[ 0 ] ) || die "Couldn't open $_[ 0 ]"; ! open( LOG2, $_[ 1 ] ) || die "Couldn't open $_[ 1 ]"; $log1 = <LOG1>; $log2 = <LOG2>; --- 106,121 ---- $datestamp = $1; } ! if ( -e "$_[ 0 ].gz" ) { ! open( LOG1, "zcat $_[ 0 ].gz |" ) || die "Couldn't open $_[ 0 ]"; ! } else { ! open( LOG1, $_[ 0 ] ) || die "Couldn't open $_[ 0 ]"; ! } ! #open( LOG1, $_[ 0 ] ) || die "Couldn't open $_[ 0 ]"; ! if ( -e "$_[ 1 ].gz" ) { ! open( LOG2, "zcat $_[ 1 ].gz |" ) || die "Couldn't open $_[ 1 ]"; ! } else { ! open( LOG2, $_[ 1 ] ) || die "Couldn't open $_[ 1 ]"; ! } ! #open( LOG2, $_[ 1 ] ) || die "Couldn't open $_[ 1 ]"; $log1 = <LOG1>; $log2 = <LOG2>; Index: run_log_scripts.pl =================================================================== RCS file: /cvsroot/dlese/logs-project/run_log_scripts.pl,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -r1.2 -r1.3 *** run_log_scripts.pl 2 Oct 2003 22:52:50 -0000 1.2 --- run_log_scripts.pl 7 Oct 2003 23:11:44 -0000 1.3 *************** *** 52,56 **** foreach $i ( 0..$#mergeLogs ) { $log = "../" . $mergeLogs[ $i ] . "/" . $mergeLogs[ $i ] . ".${month}.log"; ! if ( -e $log ) { $logs .= $log . " "; } --- 52,56 ---- foreach $i ( 0..$#mergeLogs ) { $log = "../" . $mergeLogs[ $i ] . "/" . $mergeLogs[ $i ] . ".${month}.log"; ! if ( ( -e $log ) || ( -e "${log}.gz" ) ) { $logs .= $log . " "; } |
|
From: Peter B. <pbu...@us...> - 2003-10-07 21:55:24
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1:/tmp/cvs-serv21211 Modified Files: monthly1 Log Message: Gratuitous change to test CVSEDITOR variable (which works) Index: monthly1 =================================================================== RCS file: /cvsroot/dlese/logs-project/monthly1,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** monthly1 7 Oct 2003 19:37:15 -0000 1.1 --- monthly1 7 Oct 2003 21:55:21 -0000 1.2 *************** *** 2,5 **** --- 2,6 ---- # # $Id$ + # #MONTHLY CRON at 12:01 a.m.: |
|
From: Peter B. <pbu...@us...> - 2003-10-07 19:45:30
|
Update of /cvsroot/dlese/logs-project
In directory sc8-pr-cvs1:/tmp/cvs-serv26056
Modified Files:
monthly3
Log Message:
monthly3 now exits if merged is already done and it's called from cron
Index: monthly3
===================================================================
RCS file: /cvsroot/dlese/logs-project/monthly3,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -r1.1 -r1.2
*** monthly3 7 Oct 2003 19:37:15 -0000 1.1
--- monthly3 7 Oct 2003 19:45:26 -0000 1.2
***************
*** 141,149 ****
Merge () {
if [ -s $L/merge/merged.$LASTMONTH.log ]; then
! echo
! Warn "Merged logs appear to already be in place. Sleeping 10"
! echo
! sleep 10
! Warn "Continuing ..."
fi
--- 141,154 ----
Merge () {
if [ -s $L/merge/merged.$LASTMONTH.log ]; then
! if [ "x$ISCRON" = "x" ]; then
! echo
! Warn "Merged logs appear to already be in place. Sleeping 10"
! echo
! sleep 10
! Warn "Continuing ..."
! else
! Warn "Merged logs are here and this is from cron. Exiting..."
! return 0
! fi
fi
|
|
From: Peter B. <pbu...@us...> - 2003-10-07 19:41:06
|
Update of /cvsroot/dlese/logs-project In directory sc8-pr-cvs1:/tmp/cvs-serv25413 Modified Files: monthlyREADME Log Message: Updated the readme contents Index: monthlyREADME =================================================================== RCS file: /cvsroot/dlese/logs-project/monthlyREADME,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -r1.1 -r1.2 *** monthlyREADME 7 Oct 2003 19:39:31 -0000 1.1 --- monthlyREADME 7 Oct 2003 19:41:02 -0000 1.2 *************** *** 11,32 **** # beating up the root servers then. # this script makes calls to weblog_rotate ! ! ! The CRS and SERC logs come in via an SSH forced command call to ! ./weblog_xfer. once the CRS and SERC logs are in, (manually) ! rename them and do DNS resolution if needed, e.g., ! ! LASTMONTH=`date -d "1 month ago" +%Y.%m` ! cd ../serc ! mv Incoming/serc.log ./Raw/serc.$LASTMONTH.log ! chattr +i ./Raw/serc.$LASTMONTH.log ! /www/webroot/apache/bin/logresolve.pl < ./Raw/serc.$LASTMONTH.log > serc.$LASTMONTH.log ! cd ../crs ! mv Incoming/filename ./Raw/crs.$LASTMONTH.log ! chattr +i ./Raw/crs.$LASTMONTH.log ! # crs logs hostnames, so no log_resolve.pl ! ! monthly3 # then, run 'monthly3' to merge and process ! # the logs using Ryan's scripts --- 11,18 ---- # beating up the root servers then. # this script makes calls to weblog_rotate ! monthly3 ! The CRS and SERC logs come in via an SSH forced command call to ! ./weblog_xfer. Once the CRS and SERC logs are in, monthly3 ! renames them and does DNS resolution if needed. It then does merging |
|
From: Peter B. <pbu...@us...> - 2003-10-07 19:39:34
|
Update of /cvsroot/dlese/logs-project
In directory sc8-pr-cvs1:/tmp/cvs-serv25101
Added Files:
monthlyREADME
Log Message:
monthlyREADME is some script documentation
--- NEW FILE ---
README
-- MONTHLY PROCESSING --
monthly1 # at 12:01 a.m. to rotate the webapp_logs
# as soon as reasonably possible
#
monthly2 # at anytime, to remake the symbolic links and
# and do DNS resolution. Should not run at
# 12:01 since the whole of MST is probably
# beating up the root servers then.
# this script makes calls to weblog_rotate
The CRS and SERC logs come in via an SSH forced command call to
./weblog_xfer. once the CRS and SERC logs are in, (manually)
rename them and do DNS resolution if needed, e.g.,
LASTMONTH=`date -d "1 month ago" +%Y.%m`
cd ../serc
mv Incoming/serc.log ./Raw/serc.$LASTMONTH.log
chattr +i ./Raw/serc.$LASTMONTH.log
/www/webroot/apache/bin/logresolve.pl < ./Raw/serc.$LASTMONTH.log > serc.$LASTMONTH.log
cd ../crs
mv Incoming/filename ./Raw/crs.$LASTMONTH.log
chattr +i ./Raw/crs.$LASTMONTH.log
# crs logs hostnames, so no log_resolve.pl
monthly3 # then, run 'monthly3' to merge and process
# the logs using Ryan's scripts
The processed logs are stored in logs/Reports/queryReports. Those are
available over the web at http://www.dpc.ucar.edu/projects/logs (which is a
symlink to the same directory). The sync process between webdev and live
must not overwrite that symlink.
|
|
From: Peter B. <pbu...@us...> - 2003-10-07 19:37:20
|
Update of /cvsroot/dlese/logs-project
In directory sc8-pr-cvs1:/tmp/cvs-serv24323
Added Files:
monthly1 monthly2 monthly3 monthly_functions weblog_xfer
Log Message:
Adding several Shell and Perl scripts used in log maintenance:
-weblog_xfer is called over ssh by remote sites to place their log files
in their Incoming folders
-monthly1, monthly2, monthly3 are scripts that are called out of cron each month
to coordinate log archive, dns resolution and merging
-monthly_functions is a library of bash functions used by the above scripts
--- NEW FILE ---
#!/bin/sh
#
# $Id: monthly1,v 1.1 2003/10/07 19:37:15 pburkholder Exp $
#MONTHLY CRON at 12:01 a.m.:
# 0. make sure the Raw/ directory is webadmin writable
# 1. copy dds_queries.log to Raw/dds_queries.YYYY.MM.log
# 2. cat /dev/null > dds_queries.log to zero out the log file
# (in the fraction of a second between 1 & 2 we may lose
# an entry, although that's inconsequential and unlikely)
# 3. chattr +i Raw/dds_queries.YYYY.MM.log
# 4. remake symlinkes
# - remove symlink dds_queries.YYYY.MM.log for last month
# - link dds_queries.YYYY.MM.log for current month to dds_queries.log
# 5. run log_resolve on
# ./Raw/dds_queries.YYYY.MM.log > ./dds_queries.YYYY.MM.log
#
# P. Burkholder, April 2003
W=/www/webroot # WEBROOT
LOGHOME=$W/logs
. $LOGHOME/bin/monthly_functions
cd $LOGHOME/webapps/dds || Die "Could not cd to $LOGHOME/webapps/dds"
proc_log() {
[ "x$ISCRON" = "x" ] && echo $PROG: copy log file to Raw and zero out log for new entries
if /bin/cp -p $log.log Raw/$log.$LASTMONTH.log; then
cat /dev/null > $log.log || Warn "$log not zeroed out"
[ "x$ISCRON" = "x" ] && echo $PROG: make raw file immutable
sudo /usr/bin/chattr +i Raw/$log.$LASTMONTH.log || Warn "chattr +i failed"
else
Warn "did not copy $log to Raw, appending...."
fi
[ "x$ISCRON" = "x" ] && echo $PROG: remake symlinks:
if [ -L $log.$LASTMONTH.log ] ; then
/bin/rm $log.$LASTMONTH.log || Warn "could not rm old symlink"
else
Warn "$log.$LASTMONTH.log is not a symlink as expected"
fi
/bin/ln -s $log.log $log.$THISMONTH.log || Warn "could not make new symlink"
[ "x$ISCRON" = "x" ] && echo $PROG: dns resolve Raw/$log.$LASTMONTH.log to $log.$LASTMONTH.log
if [ -x $LOGRESOLVE ] ; then
$LOGRESOLVE < Raw/$log.$LASTMONTH.log > $log.$LASTMONTH.log || Warn "logresolve of $log failed"
else
Warn "$LOGRESOLVE not executable"
fi
}
for log in dds_queries v1_queries; do
proc_log
done
# To check results, find recently changed files here:
find . -mtime -1 -ls
--- NEW FILE ---
#!/bin/sh
#
# $Id: monthly2,v 1.1 2003/10/07 19:37:15 pburkholder Exp $
#
# serc and crs are currently done manually
# This script should be run every month from cron
# - move log files w/o DNS resolution <server>/RAW/ and make immutable
# - resolve the IP address to DNS and save at <server>.year.mo.log
# - remake symlinks, e.g. flood.log -> flood.2003.03.log
# cron should set the env var. $ISCRON="Y" so it behaves more cautiously
# P. Burkholder, March 2003, September 2003
HOSTS="beta/beta catalog/catalog dpc/dpc dwel/dwel flood/flood nsdl/nsdl oai/oai page/page"
HOSTS="$HOSTS rd/rd swiki/swiki_dpc swiki/swiki_dlese training/training v1/v1_dlese v1/v1_dpc"
TESTHOSTS="bogus/swiki_dpc bogus/swiki_dlese"
W=/www/webroot # WEBROOT
LOGHOME=$W/logs
. $LOGHOME/bin/monthly_functions
Usage() {
echo usage: monthly2 -a\|-T \| -d \"server1 ...\" \| -l \"server2 ...\" \| -b \"server3 ...\"
echo " -a: automatic for HOSTS=[ $HOSTS ]"
echo " -d: do dns resolution"
echo " -l: remake symlinks"
echo " -b: do both";
echo " -T: (test) automatic for HOSTS=[ $TESTHOSTS ]"
exit 1
}
Archive() {
# Move current log file to dir/Raw and mark immutable.
# called with an argument like "swiki/swiki_dpc"
test -f $LOGHOME/$dir/$oldlog || return 0 #no old log, nothing to do
if mv $oldlog Raw/ ; then
## make immutable. For unattended operation, requires a /etc/sudoers
## entry like: "webadmin ALL=NOPASSWD: /usr/bin/chattr"
sudo /usr/bin/chattr +i Raw/$oldlog || Warn "Cannot chattr +i $oldlog"
else
Warn "Cannot mv $oldlog to Raw"
fi
return 0
}
Dns() {
if [ -s Raw/$oldlog ]; then # file exists and is non-zero size
if [ -s $oldlog ] ; then
Warn "$host/$oldlog exists. Skipping DNS if this is a cron job"
[ -n "$ISCRON" ] && return -1
Warn "This is not a cronjob. Doing DNS resolution ..."
fi
[ "x$ISCRON" = "x" ] && echo -n "LOGRESOLVE of $oldlog starting..."
if [ -x $LOGRESOLVE ] ; then
$LOGRESOLVE < Raw/$oldlog > $oldlog || Warn "Cannot do logresolve"
else
Warn "$LOGRESOLVE not executable"
fi
[ "x$ISCRON" = "x" ] && echo "...done"
else
Warn "Raw/$oldlog not found. Refusing to do DNS resolution"
return -1
fi
return 0
}
Link() {
cd $LOGHOME/$dir || return -1 #oops, move on
for type in log err; do
if [ -L $host.$type ]; then
rm $host.$type || Warn "could not rm old symlink: $host.$type"
else
Warn "$host.$type is not a symlink as expected"
fi
ln -s $host.$THISMONTH.$type $host.$type || Warn "could not make new symlink: $host.$type"
done
return 0
}
########################################################################
# MAIN
########################################################################
if [ $# -lt 1 ]; then
Usage;
fi
while getopts Tab:d:l: options 2>/dev/null;
do
case $options in
b) BOTH=$OPTARG;;
d) DNS=$OPTARG;;
l) LINK=$OPTARG;;
a) AUTO="YES";;
T) HOSTS="$TESTHOSTS"
AUTO="YES";;
\?) Usage;;
esac
done
if [ "$AUTO" == "YES" ]; then
echo "$PROG: called to do $HOSTS"
sleep 2
for dirhost in $HOSTS; do
echo "$PROG: working on $dirhost"
dir=`echo $dirhost | awk -F/ '{print $1}'` #e.g. "swiki"
host=`echo $dirhost | awk -F/ '{print $2}'` #e.g. "swiki_dpc"
oldlog=$host.$LASTMONTH.log
dirgid=`stat -t $LOGHOME/$dir | awk '{print $6}'`
if [ "$dirgid" != "$WEBADMINGID" ]; then
Warn "$dir GID is not equal $WEBADMINGID. Skipping $dirhost..."
continue
fi
if cd $LOGHOME/$dir ; then
Archive || Warn "... Archive of $dirhost skipped"
Dns || Warn "... DNS resolution of $dirhost skipped"
Link || Warn "... Link updating of $dirhost skipped"
else
Warn "could not cd to $LOGHOME/$dir, skipping..."
fi
done
exit 0
fi
if [ "x$DNS" != "x" -o "x$BOTH" != "x" ]; then
for i in $DNS $BOTH; do
echo Working on: $i
Archive $i || Warn "... Archive of $i skipped"
Dns $i || Warn "... DNS resolution of $i skipped"
done
fi
if [ "x$LINK" != "x" -o "x$BOTH" != "x" ]; then
for dirhost in $LINK $BOTH; do
dir=`echo $dirhost | awk -F/ '{print $1}'` #e.g. "swiki"
host=`echo $dirhost | awk -F/ '{print $2}'` #e.g. "swiki_dpc"
Link $dirhost || Warn "... Link updating of $i skipped"
done
fi
--- NEW FILE ---
#!/bin/bash
#
# $Id: monthly3,v 1.1 2003/10/07 19:37:15 pburkholder Exp $
#
# Last of the monthly scripts
#
# Status(CRS)
# case $? in
# 0) ;;
# 1) prepCRS
# 2) nagCRS
# *) warn
#
# Status(SERC)
# case $? in
# 0) ;;
# 1) prepSERC
# 2) nagSERC
# 3) mvSERC
# *) warn
#
# if Status(SERC) and Status(CRS)
# runMerge
# else
# nagPeter
#
W=/www/webroot # WEBROOT
LOGHOME=$W/logs
L=$LOGHOME
. $LOGHOME/bin/monthly_functions
Status() {
# 0: all ready to go
# 1: need to prep
# 2: not found
service=$1
# 1: if the intended log file is at the top level, all is well
if [ -s $L/$service/$service.$LASTMONTH.log ]; then
Warn "resolved log file $service.$LASTMONTH.log already in place"
return 0
fi
# 2: is the log file in Incoming?
case $service in
"crs") if [ -s $L/$service/Incoming/crs$CRSMONTH.log ]; then
return 1
else
return 2
fi
;;
"serc") if [ -s $L/$service/Incoming/serc.$LASTMONTH.log ]; then
return 1
elif [ -s $L/$service/Incoming/serc.log ]; then
return 3
else
return 2
fi
;;
*) return 4
;;
esac
}
mvSERC() {
# How do we know if it's really current? grep!
if grep /${M[$LMMON]}\/$LMYEAR: $L/$service/Incoming/serc.log \
1>/dev/null 2>/dev/null ; then
/bin/mv $L/$service/Incoming/serc.log \
$L/$service/Incoming/serc.$LASTMONTH.log && return 0
fi
Warn "serc.log is present, but can't mv or wrong month"
return 2
}
nagCRS() {
cat <<ENDCRS | /bin/mail -s "Automatic message from DPC logs" $CRS_EMAIL
Hi $CRS_NAME
The expected log file, crs$CRSMONTH.log, was not found at this
site. If we're mistaken, please contact $ADMIN_NAME
at $ADMIN_EMAIL
Regards,
Cron
ENDCRS
}
nagSERC() {
cat <<ENDSERC | /bin/mail -s "Automatic message from DPC logs" $SERC_EMAIL
Hi $SERC_NAME
The expected log file, serc.$LASTMONTH.log, was not found at this
site. If we're mistaken, please contact $ADMIN_NAME
at $ADMIN_EMAIL
Regards,
Cron
ENDSERC
}
prep() {
service=$1
case $service in
"serc")
/bin/mv $L/$service/Incoming/serc.$LASTMONTH.log $L/$service/Raw/
sudo chattr +i $L/$service/Raw/serc.$LASTMONTH.log
$LOGRESOLVE < $L/$service/Raw/$service.$LASTMONTH.log > $L/$service/$service.$LASTMONTH.log
if [ -s $L/$service/$service.$LASTMONTH.log ]; then
return 0
else
Warn "logresolve did not make a good $service.$LASTMONTH.log"
return -1
fi
;;
"crs")
/bin/mv $L/$service/Incoming/crs$CRSMONTH.log $L/$service/Raw/crs.$LASTMONTH.log
sudo chattr +i $L/$service/Raw/crs.$LASTMONTH.log
/bin/cp $L/$service/Raw/crs.$LASTMONTH.log $L/$service
if [ -s $L/$service/$service.$LASTMONTH.log ]; then
return 0
else
Warn "prep did not make a good $service.$LASTMONTH.log"
return -1
fi
;;
*)
Warn "Unrecognized service: $service"
return -1
;;
esac
}
Merge () {
if [ -s $L/merge/merged.$LASTMONTH.log ]; then
echo
Warn "Merged logs appear to already be in place. Sleeping 10"
echo
sleep 10
Warn "Continuing ..."
fi
export LOG_SCRIPTS=$LOGHOME/bin
export PATH=${PATH}:${LOG_SCRIPTS}
cd $LOG_SCRIPTS
echo LASTMONTH: $LASTMONTH
$L/bin/run_log_scripts.pl $LASTMONTH -m
$L/bin/run_log_scripts.pl $LASTMONTH -sx
$L/bin/run_log_scripts.pl $LASTMONTH -hx
$L/bin/run_log_scripts.pl $LASTMONTH -rx
}
#MAIN:
Status crs
case $? in
0) ;;
1) prep crs;;
2) nagCRS;;
*) Warn "Unrecognized status from Status crs";;
esac
Status serc
case $? in
0) ;;
1) prep serc;;
2) nagSERC;;
3) mvSERC && prep serc;;
*) Warn "Unrecognized status from Status serc";;
esac
if ( Status serc ) && ( Status crs ); then
Merge
fi
exit 0
--- NEW FILE ---
# Used by the monthly? scripts:
# $Id: monthly_functions,v 1.1 2003/10/07 19:37:15 pburkholder Exp $
#
# This should be sourced after setting a path
# W=/www/webroot # WEBROOT
# LOGHOME=$W/logs
# . $LOGHOME/bin/monthly_functions
#
LOGRESOLVE=/usr/local/bin/logresolve.pl
WEBADMINGID=1779
Die() {
echo $PROG: $@
exit -1
}
Warn() {
echo $PROG: $@
}
# use last month, this month unless those have already been
# set as env. variables for testing or running old scripts, e.g.:
# export LASTMONTH=`date +%Y.%m` # in 2003.04 form
# export THISMONTH=`date -d "next month" +%Y.%m` # in 2003.03 form
LASTMONTH=${LASTMONTH:=`date -d "1 month ago" +%Y.%m`}
THISMONTH=${THISMONTH:=`date +%Y.%m`}
CRSMONTH=${CRSMONTH=`date -d "1 month ago" +%Y%m`}
LMYEAR=`echo $LASTMONTH| cut -c1-4`
# trim any leading zeros
LMMON=`echo $LASTMONTH| cut -c6-7 | sed 's/^0//' `
PROG=`basename $0`
M[1]='Jan'
M[2]='Feb'
M[3]='Mar'
M[4]='Apr'
M[5]='May'
M[6]='Jun'
M[7]='Jul'
M[8]='Aug'
M[9]='Sep'
M[10]='Oct'
M[11]='Nov'
M[12]='Dec'
CRS...@ld...
CRS_NAME="Bob"
SER...@ca...
SERC_NAME="Sean"
ADMIN_NAME="Peter Burkholder"
ADM...@uc...
--- NEW FILE ---
#!/usr/bin/perl -w
# from wget http://www.snailbook.com/faq/scp-wrapper.txt
# w/ mods by Peter Burkholder, March 2003
use Digest::MD5;
# set root path R
$relpath=$ARGV[0];
$R="/www/webroot/logs/$relpath";
$now=`/bin/date +%Y.%m.%d.%H.%M`;
chomp $now;
# Since this script is called as a forced command, need to get the
# original scp command given by the client.
if ( $ENV{SSH_ORIGINAL_COMMAND} ) {
$command = $ENV{SSH_ORIGINAL_COMMAND};
} else {
$command = $now;
}
# Now, unset the ENV for safety
%ENV = ();
# Split the command string to make an argument list, and use the first
# element
@scp_argv = split /[ \t]+/, $command;
$tainted_target = shift @scp_argv;
# make some clean up of the supplied name:
($target = $tainted_target) =~ tr/0-9a-zA-z_+-./_/cs;
$target =~ s/\\//g;
########################################################################
########################################################################
### DO NOT CHANGE THE FORMAT OF THE PRINT STATEMENTS, SINCE REMOTE SITE
### WILL USE THOSE FIELDS FOR CHECKING TRANSFER SUCCESS
########################################################################
########################################################################
print "weblog_xfer: $target ";
open T, ">/$R/$target" or die "cannot write to $relpath/$target";
while (<STDIN>) {
print T or die "cannot print to filehandle T";
}
close T or die "failed closing filehandle T";
### MD5 Sum ###
open T, "</$R/$target" or die "cannot read from $relpath/$target";
binmode(T);
print Digest::MD5->new->addfile(*T)->hexdigest, "\n";
exit 0;
|