You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-27 - 17:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed May 27, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
573329879,8sleep20-21swapper/207:09:042
566428969,9sleep00-21swapper/007:08:100
569228766,6sleep10-21swapper/107:08:311
550128572,9sleep30-21swapper/307:06:043
589399260,26cyclictest0-21swapper/011:18:010
590299250,24cyclictest0-21swapper/309:49:373
590099250,25cyclictest0-21swapper/210:39:322
5896992421,2cyclictest10062-21ntp_states12:10:201
5900992320,2cyclictest994-21users09:10:272
5900992320,2cyclictest31734-21apache_processe11:45:112
5900992320,2cyclictest30614-21ntp_states11:05:182
5900992320,2cyclictest23370-21apache_accesses11:40:102
5900992320,2cyclictest22590-21df_inode11:20:152
5900992320,2cyclictest20933-21apache_volume11:00:112
5900992320,2cyclictest17247-21perl09:40:152
5900992320,2cyclictest16768-21users11:35:262
5900992320,2cyclictest16659-21diskstats11:55:152
5896992320,2cyclictest7494-21irqstats11:30:191
5896992320,2cyclictest30910-21munin-run09:10:001
5896992320,2cyclictest10230-21hddtemp_smartct09:55:171
5893992320,2cyclictest7233-21perl11:30:170
5893992320,2cyclictest20927-21apache_processe11:00:100
5893992320,2cyclictest10961-21diskstats10:15:140
5893992311,6cyclictest8865-21munin-run09:55:000
589399230,20cyclictest0-21swapper/009:49:300
5902992220,1cyclictest7095-21diskstats11:30:163
5902992220,1cyclictest4174-21apache_accesses10:50:103
5902992220,1cyclictest23964-21munin-run12:00:003
5902992219,2cyclictest25238-21ntp_states09:25:213
5900992220,1cyclictest7670-21apache_accesses11:50:112
5900992220,1cyclictest30508-21apache_processe11:25:132
5900992220,1cyclictest14107-21df11:15:142
5900992219,2cyclictest8743-21apache_volume12:10:112
5900992219,2cyclictest537-21snmpd12:03:222
5900992219,2cyclictest4249-21ntp_states10:30:192
5900992219,2cyclictest3741-21munin-run10:50:002
5900992219,2cyclictest28344-21ntp_states10:25:202
5900992219,2cyclictest23908-21apache_processe09:25:132
5896992220,1cyclictest996-21apache_volume09:50:121
5896992220,1cyclictest18678-21apache_processe10:20:121
5896992219,2cyclictest25009-21munin-run12:20:001
5896992219,2cyclictest23947-21users11:20:261
5893992220,1cyclictest27039-21apt-get10:25:130
5893992219,2cyclictest13462-21hddtemp_smartct10:55:170
589399220,5cyclictest0-21swapper/011:44:560
5902992119,1cyclictest6572-21ntp_states11:10:203
5902992119,1cyclictest17674-21apache_accesses10:00:123
5902992118,2cyclictest17080-21apache_processe12:15:103
5902992118,2cyclictest1425-21apache_accesses12:25:113
5902992118,2cyclictest1066-21ntp_states09:30:203
5902992117,2cyclictest7936-21df_inode09:15:143
5900992119,1cyclictest3943-21users10:10:252
5900992119,1cyclictest26026-21apache_accesses10:05:102
5900992118,2cyclictest9789-21apache_processe12:30:112
5900992118,2cyclictest25534-21apache_volume12:20:122
5900992118,2cyclictest11195-21hddtemp_smartct10:15:162
5896992119,1cyclictest1420-21apache_accesses12:25:121
5893992118,2cyclictest16019-21apache_volume11:55:110
589399211,19cyclictest553-21sshd08:50:140
5902992017,2cyclictest18356-21apache_accesses12:35:113
5900992018,1cyclictest4196-21apache_volume10:50:102
5900992017,2cyclictest8644-21ntp_states09:15:202
5900992017,2cyclictest370-21df_inode09:30:152
5896992018,1cyclictest893-21sensors_fan11:45:211
5896992018,1cyclictest8731-21df_inode09:35:141
5896992018,1cyclictest19662-21apache_processe10:40:111
5896992017,2cyclictest18907-21df12:35:151
589699200,19cyclictest0-21swapper/112:06:021
5893992018,1cyclictest7950-21diskstats09:15:140
5893992017,2cyclictest16463-21apache_processe09:40:100
590299190,16cyclictest0-21swapper/309:20:123
5900991916,2cyclictest15613-21apache_accesses09:20:112
590099190,19cyclictest0-21swapper/212:08:552
5896991917,1cyclictest21566-21df_inode11:00:141
5896991917,1cyclictest21383-21munin-run11:20:001
589699190,1cyclictest0-21swapper/111:35:241
589399195,13cyclictest13-21ksoftirqd/007:20:130
5893991913,5cyclictest1901-21/usr/sbin/munin10:10:140
5902991817,1cyclictest0-21swapper/312:33:523
5902991815,2cyclictest32466-21irqstats09:10:193
590299180,2cyclictest0-21swapper/311:01:123
5900991816,1cyclictest2317-21ntp_states09:50:192
5900991816,1cyclictest17679-21apache_accesses10:00:112
5896991816,1cyclictest3445-21df10:30:131
589399182,1cyclictest141rcu_preempt09:28:100
5893991814,3cyclictest30914-21diskmemload11:38:450
589399180,18cyclictest0-21swapper/012:14:320
590299172,1cyclictest0-21swapper/311:37:313
590299170,1cyclictest0-21swapper/311:45:123
5900991714,2cyclictest9413-21ntp_states09:35:192
5896991715,1cyclictest27674-21diskstats10:25:151
5893991716,1cyclictest0-21swapper/010:30:110
5902991616,0cyclictest0-21swapper/312:23:343
5902991613,2cyclictest17516-21irqstats09:40:173
590299160,16cyclictest0-21swapper/310:41:063
5900991614,1cyclictest9350-21apache_accesses09:55:122
5896991615,1cyclictest0-21swapper/111:58:171
5896991615,1cyclictest0-21swapper/110:39:321
5896991614,1cyclictest9801-21apache_volume12:30:121
589699160,16cyclictest0-21swapper/111:42:181
589699160,14cyclictest0-21swapper/111:07:141
5893991614,1cyclictest15646-21apt-get09:20:120
5893991612,3cyclictest0-21swapper/011:25:130
5893991611,5cyclictest13-21ksoftirqd/010:45:420
589399160,3cyclictest0-21swapper/009:13:070
5902991510,5cyclictest39-21ksoftirqd/312:14:023
590299150,4cyclictest0-21swapper/310:23:363
590299150,1cyclictest0-21swapper/311:20:163
5900991512,2cyclictest25763-21hddtemp_smartct09:45:162
589699150,1cyclictest0-21swapper/109:32:121
589399159,5cyclictest30914-21diskmemload11:06:510
5893991514,1cyclictest0-21swapper/008:55:140
5893991513,1cyclictest30914-21diskmemload10:20:260
5893991512,3cyclictest161rcuc/010:53:040
5893991511,3cyclictest0-21swapper/012:05:570
5893991510,2cyclictest0-21swapper/007:15:140
5902991414,0cyclictest0-21swapper/310:36:193
5902991411,2cyclictest9682-21users11:50:253
5902991411,2cyclictest24440-21apt12:00:113
590299140,1cyclictest0-21swapper/310:12:423
5900991411,2cyclictest20287-21df_inode10:40:142
589699140,0cyclictest0-21swapper/109:22:021
589399145,8cyclictest0-21swapper/012:30:110
589399144,9cyclictest0-21swapper/010:02:020
589399141,1cyclictest0-21swapper/012:36:170
5893991411,2cyclictest8389-21wc07:15:010
5893991411,2cyclictest12545-21tail10:35:190
5893991411,2cyclictest10400-21irqstats09:55:190
5893991410,3cyclictest30914-21diskmemload09:30:180
5893991410,2cyclictest30914-21diskmemload12:25:030
590299133,4cyclictest225-21systemd-journal07:29:593
5902991310,2cyclictest30285-21tr08:00:143
5902991310,2cyclictest23283-21grep08:50:263
5902991310,2cyclictest18247-21grep08:40:253
5902991310,2cyclictest10638-21tr10:15:133
590299130,13cyclictest0-21swapper/311:42:183
5900991311,1cyclictest12568-21apache_volume10:55:122
5900991310,2cyclictest28708-21grep07:55:222
5900991310,2cyclictest22048-21cat07:40:252
5900991310,2cyclictest11586-21cron08:30:002
590099130,12cyclictest23214-21cat08:50:252
589699139,3cyclictest23556-21grep07:45:161
5896991310,2cyclictest21959-21cat07:40:241
5896991310,2cyclictest20012-21ssh10:59:401
589399139,3cyclictest8199-21tail08:20:190
589399139,3cyclictest19962-21cut08:45:160
589399139,2cyclictest30914-21diskmemload11:50:310
5893991311,2cyclictest13-21ksoftirqd/011:10:120
589399131,11cyclictest0-21swapper/011:20:120
5893991310,2cyclictest18537-21grep12:15:200
5893991310,2cyclictest13347-21grep07:25:140
590299129,2cyclictest8847-21cron09:55:003
590299129,2cyclictest6888-21sed07:10:173
590299129,2cyclictest4833-21grep10:30:243
590299129,2cyclictest4760-21cron11:10:003
590299129,2cyclictest26862-21head10:05:153
590299129,2cyclictest2471-21apt10:30:003
590299129,2cyclictest20823-21sed08:45:263
590299129,2cyclictest18267-21tr07:35:143
590299129,2cyclictest17289-21tail07:30:263
590299129,2cyclictest1499-21cat08:05:263
590299129,2cyclictest14447-21sh10:55:263
590299120,11cyclictest5933-21head08:15:213
590299120,11cyclictest10693-21chrt09:55:213
590099129,2cyclictest31638-21grep08:00:262
590099129,2cyclictest18403-21ntp_states12:15:192
590099129,2cyclictest16894-21grep08:40:142
590099129,2cyclictest14473-21sed07:25:222
590099129,1cyclictest141rcu_preempt08:15:212
590099128,3cyclictest27405-21cat09:00:202
590099128,3cyclictest23881-21cat07:45:182
590099122,6cyclictest19349-21fschecks_count10:20:162
5900991210,1cyclictest12111-21grep08:30:142
590099120,1cyclictest0-21swapper/211:12:372
590099120,11cyclictest20372-21sed12:35:262
589699129,2cyclictest5068-21grep10:50:161
589699129,2cyclictest32448-21tr08:05:141
589699129,2cyclictest31736-21cut09:10:151
589699129,2cyclictest29724-21sleep08:00:001
589699129,2cyclictest25454-21tr07:50:141
589699129,2cyclictest25144-21tr09:45:131
589699129,2cyclictest20148-21awk07:40:001
589699129,2cyclictest17859-21grep09:40:201
589699129,2cyclictest11157-21head10:15:161
589699128,4cyclictest12938-21ssh11:52:591
589699126,6cyclictest25-21ksoftirqd/110:05:311
5896991211,1cyclictest0-21swapper/111:12:371
5896991211,1cyclictest0-21swapper/108:00:131
5896991210,2cyclictest0-21swapper/109:28:531
5896991210,1cyclictest38150irq/131-snd_hda07:15:141
589699120,1cyclictest0-21swapper/109:17:071
589399129,2cyclictest5331-21cut08:15:160
589399129,2cyclictest31881-21grep11:45:130
589399129,2cyclictest30460-21grep09:05:250
589399129,2cyclictest25528-21wc12:00:170
589399129,2cyclictest15010-21grep08:35:180
589399129,1cyclictest29688-21taskset12:22:080
589399126,4cyclictest13-21ksoftirqd/010:40:050
589399126,4cyclictest13-21ksoftirqd/009:37:150
589399126,2cyclictest0-21swapper/008:42:040
589399123,4cyclictest0-21swapper/007:40:120
590299118,2cyclictest376-21rm12:05:133
590299118,2cyclictest30841-21grep11:25:153
590299118,2cyclictest30039-21awk10:45:253
590299118,2cyclictest26701-21grep09:00:153
590299118,2cyclictest2376-21cut08:10:133
590299118,2cyclictest16652-21ssh11:16:163
590299118,2cyclictest15455-21tr08:35:233
590299118,2cyclictest12785-21chrt07:22:153
590299118,2cyclictest10216-21grep08:25:173
590299110,3cyclictest0-21swapper/309:37:583
590099119,1cyclictest18738-21grep07:35:182
590099118,2cyclictest9221-21sh08:25:002
590099118,2cyclictest5248-21ssh12:26:502
590099118,2cyclictest29580-21tail09:05:162
590099118,2cyclictest2571-21tr08:10:152
590099118,2cyclictest25053-21cat08:55:202
590099118,2cyclictest14197-21taskset08:35:092
589699119,1cyclictest30525-21apache_volume11:25:141
589699119,1cyclictest21311-21cron08:50:001
589699119,1cyclictest17363-21grep08:40:161
589699118,2cyclictest15938-21tr07:30:141
589699118,2cyclictest11179-21tr07:20:141
589699117,4cyclictest21645-21apache212:00:181
589699110,2cyclictest0-21swapper/110:11:151
589399119,2cyclictest161rcuc/009:00:160
589399118,2cyclictest31909-21taskset08:02:350
589399118,2cyclictest27162-21chrt07:52:540
589399118,2cyclictest26898-21cut10:05:160
589399118,2cyclictest12567-21tail08:30:170
589399115,4cyclictest13-21ksoftirqd/008:09:470
5893991110,1cyclictest0-21swapper/007:45:120
590299108,1cyclictest28576-21sed07:55:213
590299108,1cyclictest222622sleep307:40:583
590299107,2cyclictest8798-21tr07:15:153
590299107,2cyclictest25242-21sed08:55:233
590299107,2cyclictest12514-21fschecks_time08:30:163
590299100,9cyclictest27133-21chrt07:52:323
590099108,1cyclictest9477-21cat07:15:202
590099108,1cyclictest17459-21seq07:31:172
590099107,2cyclictest8224-21cat11:30:262
590099107,2cyclictest6747-21fschecks_count07:10:152
590099107,2cyclictest332-21expr08:05:152
590099107,2cyclictest12320-21wc07:20:242
590099101,3cyclictest0-21swapper/207:50:132
590099100,9cyclictest19543-21expr08:45:142
589699108,2cyclictest18128-21cat10:00:141
589699108,1cyclictest22807-21sh08:50:201
589699108,1cyclictest14536-21cat08:35:171
589699108,1cyclictest11378-21chrt08:27:241
589699107,2cyclictest8253-21perf07:15:001
589699107,2cyclictest25346-21chrt12:20:051
589699101,8cyclictest7856-21gltestperf08:20:161
589699100,9cyclictest3626-21ssh10:49:561
589699100,9cyclictest15332-21taskset07:29:301
589399108,1cyclictest27870-21cut07:55:170
589399107,2cyclictest9554-21grep08:25:150
589399107,2cyclictest16319-21cut07:30:160
59029997,1cyclictest7128-21grep08:20:123
59029997,1cyclictest28818-21cut09:05:133
59029996,2cyclictest23098-21tr07:45:143
58969998,1cyclictest5116-21cat08:15:141
58969998,1cyclictest24044-21basename08:55:141
58969997,1cyclictest3115-21grep08:10:161
58969995,3cyclictest26631-21cpuspeed_turbos09:00:151
58939990,6cyclictest13-21ksoftirqd/007:35:140
58939990,1cyclictest0-21swapper/008:13:180
58969987,1cyclictest12194-21cat08:30:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional