You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 13:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Fri Sep 11, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39702177102,28sleep10-21swapper/119:09:081
38122177102,27sleep20-21swapper/219:07:052
39122162104,29sleep30-21swapper/319:08:223
38322161100,30sleep00-21swapper/019:07:180
46942660,1sleep10-21swapper/100:09:201
293002640,5sleep3331rcuc/323:15:003
290392610,3sleep1211rcuc/122:50:221
253202590,1sleep00-21swapper/000:00:160
204042590,4sleep120403-21awk00:20:231
118612560,1sleep20-21swapper/222:37:272
19982520,8sleep334-21ksoftirqd/321:41:133
35522510,1sleep10-21swapper/121:18:481
35522510,1sleep10-21swapper/121:18:481
301552510,4sleep330148-21apt23:40:003
261292500,4sleep323785-21diskmemload21:35:213
53062480,4chrt5303-21ssh21:20:153
295422470,1sleep30-21swapper/321:14:053
107582470,4sleep10-21swapper/123:49:411
226202430,1sleep30-21swapper/322:45:223
226202430,1sleep30-21swapper/322:45:223
101342430,2sleep1211rcuc/119:20:231
274272330,2chrt0-21swapper/221:35:552
431299320,24cyclictest31451-21ssh22:04:002
431299320,24cyclictest31451-21ssh22:04:002
55982310,9sleep228-21ksoftirqd/219:10:252
431299305,15cyclictest26438-21sh22:48:402
431299305,15cyclictest26438-21sh22:48:402
431299294,11cyclictest5323-21rm21:44:352
431299291,25cyclictest9201-21diskstats19:20:142
431299291,18cyclictest19992-21perl21:55:152
431299291,16cyclictest26230-21taskset20:00:122
431299291,11cyclictest13900-21ls00:15:232
302112290,2sleep10-21swapper/100:04:061
87662280,2sleep00-21swapper/000:35:470
431399281,25cyclictest0-21swapper/319:20:503
431299282,8cyclictest5980-21ssh22:57:152
431199281,25cyclictest0-21swapper/119:40:561
431099281,25cyclictest0-21swapper/019:34:300
431399271,24cyclictest0-21swapper/320:40:403
431399271,24cyclictest0-21swapper/320:12:243
431299276,8cyclictest11054-21sh21:24:362
431299274,10cyclictest31009-21sh22:27:342
431299273,10cyclictest3554-21sh23:44:092
431299272,16cyclictest0-21swapper/221:46:522
431299271,13cyclictest0-21swapper/223:05:212
431199271,24cyclictest0-21swapper/120:29:351
431199271,24cyclictest0-21swapper/120:13:521
431199271,24cyclictest0-21swapper/120:01:521
431199271,24cyclictest0-21swapper/119:51:121
431199271,24cyclictest0-21swapper/119:26:081
431199271,1cyclictest0-21swapper/119:55:481
4310992723,2cyclictest0-21swapper/019:20:160
431099271,2cyclictest0-21swapper/020:12:240
431099271,2cyclictest0-21swapper/019:19:450
431099271,25cyclictest0-21swapper/019:39:030
431099271,24cyclictest0-21swapper/020:40:510
431099271,24cyclictest0-21swapper/019:51:150
431099271,24cyclictest0-21swapper/019:40:560
431399261,23cyclictest0-21swapper/320:39:303
431399261,23cyclictest0-21swapper/320:01:543
431399261,23cyclictest0-21swapper/319:40:563
431399261,23cyclictest0-21swapper/319:30:423
431299264,7cyclictest1758-21sh00:30:272
431299264,6cyclictest20022-21sh23:31:312
431299264,6cyclictest20022-21sh23:31:312
431299264,6cyclictest10800-21sh00:13:472
431299263,8cyclictest648-21ntpd00:29:312
4312992622,2cyclictest0-21swapper/219:20:012
431299262,17cyclictest31534-21diskmemload00:05:122
431299262,12cyclictest0-21swapper/222:43:152
431299261,8cyclictest398-21apt-get22:05:122
431299261,23cyclictest0-21swapper/219:40:532
431299260,9cyclictest23785-21diskmemload21:29:392
431199262,11cyclictest25681-21fschecks_count21:35:161
4311992620,4cyclictest25735-21cron22:00:001
431199261,1cyclictest0-21swapper/121:09:161
431199261,1cyclictest0-21swapper/119:36:441
431199261,1cyclictest0-21swapper/119:30:431
431199260,24cyclictest0-21swapper/119:49:081
4310992624,1cyclictest0-21swapper/019:26:090
4310992623,2cyclictest0-21swapper/021:08:260
4310992623,2cyclictest0-21swapper/020:17:530
4310992619,5cyclictest29258-21ssh22:25:480
431099261,24cyclictest0-21swapper/020:39:300
431099261,24cyclictest0-21swapper/020:01:540
36102260,2sleep10-21swapper/121:42:521
71092250,2sleep10-21swapper/123:21:561
4313992523,1cyclictest0-21swapper/319:39:023
431399251,2cyclictest0-21swapper/319:19:453
431399251,23cyclictest0-21swapper/319:51:163
431399250,23cyclictest0-21swapper/319:49:523
431399250,1cyclictest0-21swapper/319:25:593
431299254,6cyclictest21294-21sh21:31:262
431299254,6cyclictest18944-21sh21:54:542
431299253,9cyclictest30975-21sh22:51:312
431299253,9cyclictest1414-21ssh22:30:112
431299253,7cyclictest9846-21sh00:36:522
431299253,6cyclictest7284-21sh23:46:092
431299253,6cyclictest23505-21sh00:23:092
431299253,6cyclictest23291-21sh22:21:322
431299253,6cyclictest15106-21sh23:28:182
431299253,10cyclictest24009-21sh23:59:382
4312992515,7cyclictest13472-21df_inode19:30:132
431299251,23cyclictest0-21swapper/219:26:162
431299251,22cyclictest0-21swapper/219:36:332
431299250,23cyclictest0-21swapper/220:17:372
431299250,23cyclictest0-21swapper/220:12:422
4311992523,1cyclictest0-21swapper/120:39:501
431199251,1cyclictest0-21swapper/120:23:231
4310992522,2cyclictest0-21swapper/020:58:120
431099251,8cyclictest29543-21apt-get23:15:090
4310992515,8cyclictest9932-21ssh23:48:510
431099250,24cyclictest0-21swapper/019:49:520
431099250,1cyclictest0-21swapper/019:56:560
4313992422,1cyclictest0-21swapper/321:08:263
4313992422,1cyclictest0-21swapper/320:17:533
4313992415,6cyclictest5203-21ssh22:56:293
4313992413,8cyclictest18950irq/143-ahci[0023:10:003
431399241,1cyclictest0-21swapper/319:56:563
431399240,21cyclictest7779-21ntpq00:35:223
431299243,9cyclictest15266-21sh23:04:532
431299243,6cyclictest28046-21sh21:12:332
431299243,6cyclictest11827-21sh22:13:282
4312992422,1cyclictest0-21swapper/220:40:342
431299241,9cyclictest3863-21ssh21:19:062
431299241,9cyclictest3863-21ssh21:19:062
431299240,22cyclictest0-21swapper/219:50:452
431299240,10cyclictest23785-21diskmemload23:50:142
4311992422,1cyclictest0-21swapper/120:40:381
4311992413,7cyclictest32454-21basename22:05:001
4311992413,7cyclictest32454-21basename22:05:001
431199240,1cyclictest0-21swapper/120:56:201
431099242,12cyclictest15898-21ntp_states22:40:210
4310992421,2cyclictest0-21swapper/020:23:140
431099241,7cyclictest9211-21ntp_states22:35:220
4310992411,3cyclictest12-21ksoftirqd/023:00:010
260722240,2chrt0-21swapper/022:24:200
85582230,1sleep00-21swapper/021:22:040
4313992321,1cyclictest0-21swapper/320:58:123
431399231,6cyclictest25874-21grep22:00:003
4312992321,1cyclictest0-21swapper/221:09:142
4312992321,1cyclictest0-21swapper/220:57:232
4312992321,1cyclictest0-21swapper/220:30:042
4312992321,1cyclictest0-21swapper/220:29:452
431299231,8cyclictest23785-21diskmemload23:37:192
4312992314,7cyclictest10114-21cron23:25:002
431299230,7cyclictest16668-21ssh22:16:372
4311992318,3cyclictest28536-21cstates20:05:141
4311992317,4cyclictest22408-21sh23:10:011
431099233,12cyclictest16049-21ssh23:05:120
4310992315,6cyclictest989-21ls21:40:260
4310992313,7cyclictest32251-21python321:15:240
4310992313,7cyclictest32251-21python321:15:240
431399224,4cyclictest141rcu_preempt00:15:173
4313992220,1cyclictest0-21swapper/320:23:143
431399221,6cyclictest29962-21users22:50:303
4313992214,6cyclictest7985-21timerwakeupswit20:30:243
4312992220,1cyclictest0-21swapper/220:39:592
4312992212,6cyclictest53050irq/142-enp3s0-23:10:042
431299220,7cyclictest32347-21ssh23:16:282
431299220,6cyclictest25940-21ntp_states00:00:212
431199222,13cyclictest23785-21diskmemload21:30:121
431199222,13cyclictest18183-21apt-get23:55:131
431199222,13cyclictest12828-21apt-get21:50:131
431199221,19cyclictest15359-21ssh23:28:331
431099228,3cyclictest141rcu_preempt00:10:250
431099225,3cyclictest23785-21diskmemload23:23:370
431099221,13cyclictest23785-21diskmemload21:36:340
262732220,2sleep10-21swapper/123:36:051
180102220,2sleep00-21swapper/000:19:220
431399214,4cyclictest23785-21diskmemload23:45:263
431399211,17cyclictest15467-21gltestperf22:40:163
4312992112,6cyclictest29261-21ntpq20:05:212
4312992112,6cyclictest16058-21sed20:50:202
4311992113,6cyclictest30461-21cron21:15:001
4311992112,7cyclictest8437-21python322:10:231
4311992112,5cyclictest3422-21latency_hist23:20:001
4311992111,6cyclictest22698-21grep22:45:231
4311992111,6cyclictest22698-21grep22:45:231
431099219,3cyclictest191962sh23:30:420
431099219,3cyclictest191962sh23:30:410
4310992114,4cyclictest23785-21diskmemload00:33:450
4310992113,6cyclictest29123-21memory22:50:240
4310992113,5cyclictest9497-21cat23:00:150
4310992112,7cyclictest24405-40logrotate00:00:010
4310992112,6cyclictest31247-21idleruntime23:40:160
431399208,8cyclictest31531-21apt-get00:05:123
431399207,11cyclictest6692-21cron22:10:003
431399206,11cyclictest0-21swapper/323:25:153
431399205,12cyclictest640-21snmpd21:45:173
431399204,3cyclictest23785-21diskmemload23:43:253
431399204,3cyclictest23785-21diskmemload22:14:103
431399204,14cyclictest30271-21ssh23:15:153
431399203,3cyclictest23785-21diskmemload00:24:523
431399202,16cyclictest31122-21perf00:05:003
431399202,12cyclictest18331-21ssh23:30:243
431399202,12cyclictest18331-21ssh23:30:243
431399202,10cyclictest75-21kswapd023:00:113
4313992013,4cyclictest6657-21sh23:21:293
4313992011,7cyclictest13168-21which21:25:233
431399201,16cyclictest23785-21diskmemload23:58:533
431399201,16cyclictest14295-21sort20:45:233
431399200,7cyclictest23785-21diskmemload21:50:133
4312992012,6cyclictest20682-21ntp_kernel_pll_19:45:192
4312992010,7cyclictest24530-21fw_packets19:55:142
431199209,7cyclictest52850irq/140-enp3s0-00:35:141
431199209,3cyclictest26772ssh00:31:241
4311992012,5cyclictest636-21runrttasks22:43:561
4311992011,6cyclictest636-21runrttasks23:53:351
4311992011,6cyclictest278-21systemd-journal22:25:001
431199201,16cyclictest26294-21ssh00:25:171
4311992011,6cyclictest18325-21ntp_states23:30:251
4311992011,6cyclictest18325-21ntp_states23:30:241
4311992010,6cyclictest52850irq/140-enp3s0-21:45:121
431099204,3cyclictest23785-21diskmemload22:15:470
431099203,3cyclictest23785-21diskmemload22:06:450
431099203,11cyclictest7110-21apt-get22:10:110
431099202,9cyclictest24605-21unixbench_singl23:10:260
431099202,16cyclictest6433-21perf20:30:000
431099201,9cyclictest1793-21diskstats22:30:140
431099201,6cyclictest0-21swapper/021:47:400
431099201,4cyclictest0-21swapper/021:53:440
4310992012,6cyclictest405-21python300:05:230
4310992012,6cyclictest25480-21ls00:25:010
4310992012,6cyclictest20676-21readlink21:00:220
4310992012,6cyclictest11070-21tune2fs23:25:150
4310992011,7cyclictest5950-21seq19:11:270
4310992010,7cyclictest26131-21df_inode00:25:150
431399199,7cyclictest21604-21grep22:20:213
431399193,13cyclictest6407-21sed00:10:193
431399192,12cyclictest32408-21ssh00:30:113
431399192,10cyclictest23785-21diskmemload00:25:123
4313991912,5cyclictest31000-21seq22:27:343
4313991911,6cyclictest5965-21seq19:11:393
4313991911,6cyclictest27148-21grep22:00:203
4313991911,6cyclictest27148-21grep22:00:203
4313991911,5cyclictest15790-21sed20:50:173
4312991911,6cyclictest4477-21sed20:25:002
4312991911,6cyclictest20444-21ntpq21:00:192
4311991917,1cyclictest0-21swapper/120:34:031
4311991913,4cyclictest7759-21sh22:59:041
4311991912,4cyclictest23785-21diskmemload23:00:221
4311991911,6cyclictest21542-21cat21:05:001
4311991911,6cyclictest16163-21grep20:50:211
4311991911,6cyclictest14676-21sed22:15:171
431099198,9cyclictest52950irq/141-enp3s0-21:10:180
431099197,8cyclictest20673-21processes21:55:220
431099194,13cyclictest27585-21ssh22:00:240
431099194,13cyclictest27585-21ssh22:00:240
431099192,4cyclictest141rcu_preempt21:25:120
431099192,11cyclictest15250-21apt-get20:50:130
4310991913,4cyclictest21623-21cpuspeed_turbos22:45:150
4310991913,4cyclictest21623-21cpuspeed_turbos22:45:140
4310991911,6cyclictest24908-21awk23:35:200
4310991911,6cyclictest12996-21cat23:50:250
431399181,4cyclictest0-21swapper/322:37:553
4313991813,3cyclictest20639-21sendmail_mailst21:00:213
4313991813,3cyclictest14407-21idleruntime22:15:153
4313991811,5cyclictest52950irq/141-enp3s0-22:30:123
4313991810,6cyclictest6550-21date20:30:003
4313991810,6cyclictest28967-21basename20:05:183
4313991810,6cyclictest11820-21cat23:50:143
431399180,4cyclictest32339-21ls21:15:253
431399180,4cyclictest32339-21ls21:15:253
431299181,10cyclictest0-21swapper/220:45:122
431199189,7cyclictest31030-21tune2fs23:40:151
431199189,6cyclictest7006-21expr19:15:131
431199189,5cyclictest24532-21unixbench_multi23:10:251
431199189,5cyclictest11889-21ssh21:25:121
431199188,3cyclictest2876-21timerandwakeup22:30:231
431199182,12cyclictest6879-21ssh00:10:231
431199181,5cyclictest141rcu_preempt22:05:181
4311991813,3cyclictest13087-21/usr/sbin/munin20:45:151
4311991812,4cyclictest13383-21netstat00:15:181
4311991811,5cyclictest919-21cat20:15:191
4311991811,5cyclictest12833-21seq22:38:251
4311991810,6cyclictest5014-21cut19:10:191
4311991810,6cyclictest28672-21cat22:25:261
431099184,12cyclictest18562-21apt-get21:30:130
4310991811,5cyclictest29193-21needreboot20:05:200
4310991810,6cyclictest14114-21which20:45:210
431399172,4cyclictest34-21ksoftirqd/321:30:223
4311991712,3cyclictest6350-21grep21:20:241
4310991710,5cyclictest6912-21expr20:30:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional