You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-20 - 07:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Jun 20, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
80902178118,34sleep20-21swapper/219:08:222
79272176120,49sleep30-21swapper/319:06:153
63152160106,27sleep00-21swapper/019:05:070
81932157107,28sleep10-21swapper/119:09:431
122802680,2sleep012-21ksoftirqd/021:45:290
265832630,1sleep10-21swapper/123:56:451
196102590,1sleep20-21swapper/223:03:482
253512580,1sleep30-21swapper/323:55:313
104942560,2sleep20-21swapper/221:45:122
29422520,2sleep00-21swapper/000:27:420
140912520,1sleep00-21swapper/023:23:500
325242510,2sleep20-21swapper/221:36:492
314672510,8sleep012-21ksoftirqd/021:11:580
40052500,1sleep10-21swapper/122:04:281
200302490,2sleep00-21swapper/000:15:430
169722480,3sleep334-21ksoftirqd/321:25:173
129282470,1sleep10-21swapper/100:10:261
157672430,2sleep10-21swapper/123:00:231
282832420,1sleep30-21swapper/321:58:043
76832330,2chrt0-21swapper/300:06:463
848999292,18cyclictest20976-1kworker/u9:000:21:320
848999275,15cyclictest24981-21kworker/u8:023:44:460
848999272,7cyclictest27454-21ssh23:33:470
8489992724,2cyclictest131rcuc/022:13:160
8489992711,8cyclictest12-21ksoftirqd/023:19:290
849299261,24cyclictest0-21swapper/319:15:423
848999265,7cyclictest12-21ksoftirqd/023:50:100
848999263,8cyclictest12-21ksoftirqd/023:57:070
848999262,6cyclictest8364-21ssh21:43:180
8489992622,3cyclictest131rcuc/022:52:350
848999261,6cyclictest12-21ksoftirqd/021:55:230
848999259,2cyclictest141rcu_preempt21:38:020
848999254,8cyclictest12-21ksoftirqd/022:00:160
848999253,8cyclictest12-21ksoftirqd/000:37:420
848999253,6cyclictest12-21ksoftirqd/023:08:350
848999253,6cyclictest12-21ksoftirqd/000:01:510
848999253,15cyclictest2537-21ssh21:15:120
848999252,8cyclictest12-21ksoftirqd/022:31:400
8489992523,1cyclictest0-21swapper/019:15:030
8489992521,3cyclictest131rcuc/020:49:580
8489992521,3cyclictest131rcuc/019:49:340
848999250,8cyclictest12-21ksoftirqd/019:20:370
66592250,2sleep00-21swapper/000:30:200
66592250,2sleep00-21swapper/000:30:190
250512250,2sleep228-21ksoftirqd/222:43:552
192972250,3sleep319295-21ssh22:15:183
849299243,15cyclictest15710-21sh23:25:113
849299240,23cyclictest0-21swapper/319:14:433
8491992422,1cyclictest0-21swapper/219:14:362
8490992422,1cyclictest0-21swapper/119:15:331
848999249,2cyclictest141rcu_preempt22:25:550
848999248,6cyclictest13028-21sed00:10:250
848999247,8cyclictest12-21ksoftirqd/023:14:290
848999245,7cyclictest12-21ksoftirqd/021:30:160
848999245,7cyclictest12-21ksoftirqd/021:30:150
848999244,7cyclictest12-21ksoftirqd/023:49:410
848999243,8cyclictest12-21ksoftirqd/022:35:110
848999242,7cyclictest12-21ksoftirqd/022:23:130
848999242,6cyclictest12-21ksoftirqd/022:43:270
8489992422,1cyclictest0-21swapper/019:14:420
8489992421,2cyclictest131rcuc/000:07:090
8489992420,3cyclictest131rcuc/020:54:430
8489992419,3cyclictest131rcuc/020:25:450
848999241,6cyclictest12-21ksoftirqd/020:30:150
848999240,8cyclictest12-21ksoftirqd/023:00:120
848999240,8cyclictest12151-21ssh22:58:050
8491992313,7cyclictest31898-21apt-get22:25:122
8491992313,6cyclictest27083-21cut22:45:152
8490992314,7cyclictest18448-21python320:45:231
8490992314,6cyclictest23875-21df19:45:131
8490992313,7cyclictest6609-21fw_packets20:20:161
848999239,2cyclictest141rcu_preempt22:47:400
848999238,6cyclictest12-21ksoftirqd/022:09:110
848999238,2cyclictest141rcu_preempt22:18:470
848999233,7cyclictest12-21ksoftirqd/023:38:110
848999233,6cyclictest12-21ksoftirqd/021:20:500
848999232,5cyclictest12-21ksoftirqd/021:05:150
8489992320,2cyclictest131rcuc/021:25:410
8489992320,2cyclictest131rcuc/020:42:390
848999231,9cyclictest15584-21ssh23:25:050
8489992319,2cyclictest131rcuc/021:50:100
41142230,2sleep20-21swapper/223:40:172
193432230,1sleep20-21swapper/223:27:202
177042230,2sleep20-21swapper/219:30:162
155132230,8sleep122-21ksoftirqd/123:49:071
849299229,9cyclictest26922-21df22:45:143
8492992214,6cyclictest19744-21users00:15:263
8491992220,1cyclictest0-21swapper/223:34:182
8490992220,1cyclictest0-21swapper/119:14:501
8490992217,3cyclictest10386-21sh20:30:001
8490992216,4cyclictest18853-21sh00:40:001
8490992213,7cyclictest8176-21df_inode22:55:141
8490992213,7cyclictest26358-21users00:20:261
848999224,6cyclictest12-21ksoftirqd/019:55:530
8489992218,3cyclictest131rcuc/020:20:080
8489992218,3cyclictest131rcuc/019:41:090
848999220,6cyclictest3393-21taskset20:10:470
46462220,1sleep30-21swapper/323:40:213
849299212,6cyclictest141rcu_preempt22:25:003
8492992114,5cyclictest28773-21diskmemload00:22:503
8492992113,6cyclictest17475-21ssh22:38:043
8492992112,7cyclictest5626-21cat00:30:113
8492992112,7cyclictest5626-21cat00:30:103
8492992112,5cyclictest31630-21switchtime00:00:223
8492992111,6cyclictest660-21lldpd22:32:323
849199219,6cyclictest11931-21apt-get00:35:042
849199219,2cyclictest141rcu_preempt21:59:272
849199211,9cyclictest16867-21wc21:25:162
8491992119,1cyclictest0-21swapper/219:15:322
8491992114,5cyclictest28-21ksoftirqd/200:00:002
8491992113,6cyclictest9526-21users22:55:272
8491992112,6cyclictest15854-21sed20:40:172
8491992112,6cyclictest13006-21unixbench_singl00:10:252
849199211,13cyclictest28176-21sh21:10:112
8490992114,4cyclictest7368-21sh22:55:001
8490992113,6cyclictest15830-21netstat19:25:191
8490992112,7cyclictest30729-21tune2fs00:00:151
8490992112,7cyclictest29500-21swap21:10:221
8490992112,7cyclictest24119-21cut21:00:161
8490992112,7cyclictest22568-21which20:55:221
8490992112,6cyclictest28660-21irqstats23:10:171
8490992112,6cyclictest16467-21cut23:25:191
8490992111,7cyclictest32260-21df22:25:131
8490992111,6cyclictest19055-21tr00:15:201
849099210,10cyclictest270592sh21:56:511
848999212,6cyclictest12-21ksoftirqd/019:31:380
848999211,7cyclictest12-21ksoftirqd/019:50:070
8489992117,3cyclictest131rcuc/020:01:150
848999210,6cyclictest21075-21taskset19:39:210
848999210,6cyclictest16506-21taskset19:25:370
62582210,2sleep30-21swapper/322:05:223
849299208,4cyclictest141rcu_preempt21:30:223
849299208,4cyclictest141rcu_preempt21:30:223
849299202,15cyclictest27541-21latency_hist23:10:003
849299201,7cyclictest31332rm22:03:363
849299201,7cyclictest17338-21perl23:50:143
8492992013,5cyclictest3016-21awk20:10:233
8492992013,5cyclictest29471-21seq19:57:033
8492992013,5cyclictest28689-21perl21:10:153
8492992012,6cyclictest11737-21which20:30:223
849199208,3cyclictest27757-21diskmemload23:38:012
849199201,5cyclictest0-21swapper/222:14:462
849199201,5cyclictest0-21swapper/221:17:362
8491992015,3cyclictest5649-21grep00:05:212
8491992015,3cyclictest19487-21timerandwakeup00:15:252
8491992013,4cyclictest31942-21grep00:00:242
8491992012,6cyclictest1596-21fw_packets22:50:142
8491992012,4cyclictest0-21swapper/221:05:132
8490992014,4cyclictest7722-21sh21:18:551
8490992013,5cyclictest18586-21sh21:26:171
8490992012,6cyclictest29926-21date20:00:001
8490992012,6cyclictest22255-21ntpq23:05:191
8490992012,6cyclictest12432-21seq20:33:541
8490992011,7cyclictest560-21dbus-daemon21:10:001
849099201,15cyclictest5642-21apt-get00:30:121
849099201,15cyclictest5642-21apt-get00:30:111
8490992010,6cyclictest53050irq/142-enp3s0-22:15:111
848999202,6cyclictest12-21ksoftirqd/020:55:230
8489992017,2cyclictest131rcuc/020:07:280
848999201,6cyclictest12-21ksoftirqd/021:03:430
320572200,1sleep30-21swapper/323:12:493
174932200,4chrt28-21ksoftirqd/222:38:042
849299192,12cyclictest15152-21apt-get20:40:123
849299192,12cyclictest11675-21apt-get22:10:123
8492991913,4cyclictest6758-21sh23:18:143
8492991912,5cyclictest22571-21sensors19:40:223
8492991912,4cyclictest27757-21diskmemload00:11:303
8492991911,6cyclictest32404-21cut22:25:163
8492991911,6cyclictest30479-21ntpq21:35:203
8492991911,6cyclictest2775-21uptime22:50:243
8492991911,6cyclictest27275-21seq19:51:183
8492991911,6cyclictest25077-21seq19:45:313
8492991911,6cyclictest17354-21cat20:45:133
8492991911,6cyclictest1265-21ssh23:38:183
8492991910,6cyclictest18272-21python319:30:243
849299190,1cyclictest0-21swapper/319:28:113
849199198,2cyclictest0-21swapper/221:50:202
849199192,11cyclictest14975-21apt-get19:25:122
849199191,6cyclictest30653-21taskset00:24:432
849199191,5cyclictest0-21swapper/223:50:042
8491991912,5cyclictest28-21ksoftirqd/221:05:002
8491991912,5cyclictest11578-21sed23:45:232
8491991911,6cyclictest9873-21cat23:20:182
8491991911,6cyclictest8715-21expr20:25:132
8491991911,6cyclictest21323-21cron20:55:002
8491991911,6cyclictest21261-21date19:40:002
8491991911,6cyclictest11401-21memory20:30:192
8491991910,6cyclictest278-21systemd-journal23:17:002
849199190,6cyclictest4099-21/usr/sbin/munin21:40:132
849099198,9cyclictest0-21swapper/123:40:131
849099192,12cyclictest3986-21apt-get20:15:141
849099192,11cyclictest4957-21apt-get22:05:131
8490991914,3cyclictest6135-21/usr/sbin/munin22:30:181
8490991913,4cyclictest28076-21sh22:21:361
8490991913,4cyclictest13463-21sh21:46:401
8490991912,5cyclictest10147-21ls21:45:001
8490991912,4cyclictest27757-21diskmemload22:43:141
8490991911,6cyclictest4437-21seq23:15:541
8490991910,6cyclictest278-21systemd-journal20:10:001
8489991915,3cyclictest131rcuc/020:16:430
848999190,8cyclictest12-21ksoftirqd/020:35:360
299472190,2sleep10-21swapper/121:35:151
849299189,7cyclictest5073-21python321:40:233
849299186,8cyclictest29985-21apt-get20:00:113
849299186,3cyclictest141rcu_preempt22:40:243
849299185,3cyclictest27757-21diskmemload21:50:183
8492991813,3cyclictest5077-21sh21:16:153
8492991813,3cyclictest15228-21sh23:48:513
8492991813,3cyclictest14171-21sh21:47:233
8492991812,4cyclictest8441-21sh22:55:163
8492991812,4cyclictest26284-21sh23:32:363
8492991812,4cyclictest12324-21sh23:22:043
8492991811,5cyclictest9097-21basename20:25:163
8492991811,5cyclictest32223-21load00:25:183
8492991811,5cyclictest23300-21seq20:58:143
8492991811,5cyclictest15760-21sh23:00:223
8492991810,6cyclictest24180-21cut21:00:163
8492991810,6cyclictest14847-21date19:25:003
8492991810,6cyclictest13085-21fw_packets00:35:153
8492991810,6cyclictest10604-21ls21:20:233
849199188,8cyclictest31550-21seq20:01:002
849199181,8cyclictest22773-21tr21:30:112
849199181,8cyclictest22773-21tr21:30:112
849199181,5cyclictest26226-21ntp_states22:20:212
8491991813,3cyclictest13426-21sh21:22:502
8491991812,4cyclictest28-21ksoftirqd/221:00:012
849199181,12cyclictest6165-21apt-get22:30:122
849199181,12cyclictest12837-21apt-get19:20:122
8491991810,6cyclictest6925-21sed20:20:202
8491991810,6cyclictest27564-21cron19:55:002
8491991810,6cyclictest2503-21cut20:10:172
849099189,6cyclictest29970-21acpi20:00:101
849099187,6cyclictest53050irq/142-enp3s0-21:20:101
849099186,4cyclictest232332cat23:30:171
849099181,6cyclictest22683-21apt-get21:30:111
849099181,6cyclictest22683-21apt-get21:30:101
8490991813,3cyclictest22725-21timerwakeupswit19:40:251
8490991812,4cyclictest6762-21sh00:05:511
8490991812,4cyclictest32117-21sh23:36:431
8490991812,4cyclictest22-21ksoftirqd/123:25:001
8490991812,4cyclictest20391-21sh21:51:531
8490991811,5cyclictest31062-21sh22:48:111
8490991811,5cyclictest20384-21run-parts19:35:221
8490991811,5cyclictest17210-21entropy23:50:141
8490991811,4cyclictest27757-21diskmemload22:11:401
8490991811,4cyclictest12981-21sh22:35:041
849099181,12cyclictest19427-21apt-get20:50:121
8490991810,6cyclictest2997-21sensors_temp20:10:221
8490991810,6cyclictest19122-21cat19:35:021
309042180,2sleep20-21swapper/223:11:392
849299176,8cyclictest19254-21apt-get19:35:103
8492991712,3cyclictest32099-21/usr/sbin/munin20:05:113
8492991711,4cyclictest21441-21sed20:55:003
849299171,11cyclictest12991-21apt-get20:35:123
849199179,5cyclictest25011-21grep19:45:252
849199178,7cyclictest560-21dbus-daemon20:50:002
849199172,10cyclictest31374-21ssh00:25:132
849199171,7cyclictest18296-21ssh22:15:052
8491991712,3cyclictest6379-21rm00:30:162
8491991712,3cyclictest6379-21rm00:30:162
8491991712,3cyclictest4932-21/usr/sbin/munin22:05:172
8491991712,3cyclictest27803-21/usr/sbin/munin19:55:152
849199171,14cyclictest4832-21head20:15:192
8491991710,5cyclictest28-21ksoftirqd/223:05:562
8491991710,3cyclictest32254-21unixbench_multi22:00:242
849099179,6cyclictest32548-21cut00:25:211
8490991713,2cyclictest26604-21ntp_states19:50:191
8490991712,3cyclictest16061-21ntp_offset20:40:201
8490991710,5cyclictest14743-21seq20:38:271
8490991710,5cyclictest13346-21grep19:20:151
849299169,5cyclictest5629-21seq20:17:013
849299169,5cyclictest25919-21ls21:05:133
8492991612,2cyclictest6122-21/usr/sbin/munin20:20:253
849199169,5cyclictest560-21dbus-daemon20:35:002
849199168,6cyclictest922-21cat20:05:272
8491991510,3cyclictest21391-21/usr/sbin/munin19:40:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional