You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 14:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Mar 14, 2026 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
239420199901,4cyclictest150-21pr/ttyS219:45:030
2394201998629,40cyclictest2463204-21ssh23:30:060
2394201998629,40cyclictest2463204-21ssh23:30:060
2394201998529,51cyclictest2406611-21ssh21:23:440
2394201998529,51cyclictest2406611-21ssh21:23:430
239420199842,5cyclictest2451778-21munin-run23:05:010
239420199832,75cyclictest2431611-21ssh22:20:010
239420199812,42cyclictest2422736-21ssh22:00:000
2394201998018,33cyclictest2439262-21ssh22:36:420
239420199801,74cyclictest48-21kcompactd022:05:140
239420199801,49cyclictest2494089-21rm00:39:320
239420199782,46cyclictest2424944-21munin-run22:05:010
239420199782,46cyclictest2424944-21munin-run22:05:010
2394201997822,51cyclictest2484907-21ssh00:19:050
239420199751,4cyclictest2448326-21sshd22:57:020
2394201997420,44cyclictest16850irq/134-800000022:26:070
239420199731,7cyclictest2489884-21cut00:30:000
2394207997228,39cyclictest2450382-21sh23:01:351
239420199711,6cyclictest2427481-21ssh22:10:150
2394207997011,56cyclictest2400376-21diskmemload21:40:011
2394201997028,25cyclictest2486242-21ssh00:21:390
239420199702,39cyclictest2478612-21munin-run00:05:000
239420199702,39cyclictest2460061-21sshd23:23:230
239420199701,51cyclictest2419239-21ssh21:51:580
239420199701,51cyclictest2419239-21ssh21:51:570
2394201997012,53cyclictest16850irq/134-800000021:18:200
239420199701,16cyclictest2437275-21ssh22:32:210
239421599695,28cyclictest2422524-21ssh21:59:333
239420199691,64cyclictest2461735-21ssh23:26:590
239420199691,36cyclictest2432134-21ssh22:20:390
239420199691,2cyclictest0-21swapper/000:10:020
2394211996812,35cyclictest311-21rs:main2
2394201996843,21cyclictest16850irq/134-800000023:40:560
2394201996820,43cyclictest16850irq/134-800000021:48:290
2394201996819,44cyclictest2412579-21ssh21:37:050
2394207996728,35cyclictest2467387-21/usr/sbin/munin23:40:001
2394201996729,34cyclictest2400915-21ssh21:10:430
239420199672,60cyclictest2481201-21ssh00:10:240
239420199671,61cyclictest2458512-21latency_hist23:20:020
239420199671,5cyclictest2446589-21ssh22:53:200
239420199671,5cyclictest0-21swapper/023:45:390
239420199671,2cyclictest0-21swapper/021:45:020
239420199671,16cyclictest2476148-21sh23:59:300
2394201996627,23cyclictest2441156-21ssh22:40:530
2394201996610,48cyclictest2394658-21cron19:19:590
2394201996531,30cyclictest2411743-21grep21:35:020
239420199652,58cyclictest2397273-21munin-run20:10:010
2394201996519,41cyclictest2395696-21latency_hist19:40:010
2394211996454,6cyclictest2458506-21kworker/2:100:15:012
239420799644,20cyclictest2397012-21modprobe20:05:011
2394207996422,37cyclictest2483029-21sh00:14:531
2394207996422,36cyclictest1-21systemd19:25:001
2394201996437,24cyclictest16950irq/224-800000020:00:010
239420199643,55cyclictest2400342-21sshd21:06:230
2394201996421,39cyclictest2407997-21ssh21:26:350
2394215996314,10cyclictest37-21ksoftirqd/321:25:013
2394215996314,10cyclictest37-21ksoftirqd/321:25:003
239420199632,56cyclictest2399340-21munin-run20:49:590
239420199632,56cyclictest2397009-21munin-run20:05:010
239420199631,5cyclictest0-21swapper/020:35:450
2394207996221,36cyclictest2416803-21rm21:46:271
2394207996216,41cyclictest2420255-1kworker/u9:122:51:481
239420199621,51cyclictest16850irq/134-800000023:35:350
239420199621,2cyclictest0-21swapper/000:32:010
239421199612,46cyclictest2463168-21ssh23:30:022
239421199612,46cyclictest2463168-21ssh23:30:012
2394201996127,30cyclictest2398214-21sed20:25:010
239420199612,54cyclictest16850irq/134-800000023:05:240
239420199611,6cyclictest2472275-21sshd23:50:240
239420199611,3cyclictest0-21swapper/023:10:240
239420799606,47cyclictest2420460-21munin-plugin-st21:55:011
239420799606,47cyclictest2420460-21munin-plugin-st21:55:001
239420799604,47cyclictest2489976-21wc00:30:021
2394207996030,26cyclictest2408982-21sh21:29:051
2394207996013,40cyclictest23-21ksoftirqd/122:50:001
2394201996042,13cyclictest16950irq/224-800000020:20:010
2394211995949,6cyclictest2458506-21kworker/2:100:35:022
2394211995947,6cyclictest2397537-21kworker/2:021:16:572
2394207995928,27cyclictest1-21systemd00:40:011
2394207995921,33cyclictest309-21in:imuxsock20:45:031
2394207995916,38cyclictest0-21swapper/121:05:011
239420199592,53cyclictest48-21kcompactd022:47:000
239420199592,52cyclictest2394938-21munin-run19:25:010
239420199591,15cyclictest2398385-21cat20:30:020
2394215995838,15cyclictest3467602-21systemd-journal00:00:003
2394211995846,7cyclictest2458506-21kworker/2:100:35:362
2394211995846,6cyclictest2429736-21kworker/2:122:31:002
239420799586,8cyclictest2394389-21modprobe19:15:001
2394207995814,40cyclictest2432311-1kworker/u9:022:31:381
239420799581,12cyclictest2432316-21ssh22:21:051
239420799581,11cyclictest2404271-21ssh21:18:261
239420199582,51cyclictest2398569-21munin-run20:35:010
2394215995712,10cyclictest37-21ksoftirqd/322:41:533
2394211995748,6cyclictest2436153-21kworker/2:322:55:012
2394211995747,7cyclictest2451773-21kworker/2:223:37:422
239420799572,7cyclictest2460717-21munin-run23:25:021
239420799572,17cyclictest2468973-21sshd23:43:231
239420799572,16cyclictest2479747-21ssh00:07:211
2394207995712,39cyclictest275-21dbus-daemon23:15:011
239420199572,50cyclictest2399853-21munin-run21:00:010
239420199572,50cyclictest2399853-21munin-run21:00:000
239420199572,50cyclictest2396227-21munin-run19:50:000
2394215995611,8cyclictest37-21ksoftirqd/323:40:533
2394211995648,5cyclictest2436153-21kworker/2:322:57:162
2394211995647,5cyclictest2451773-21kworker/2:223:18:252
239420799568,12cyclictest23-21ksoftirqd/120:45:011
239420799564,11cyclictest2462983-21cat23:30:001
239420799563,5cyclictest0-21swapper/121:15:021
239420799563,10cyclictest2396178-21latency_hist19:45:031
239420799562,9cyclictest2453655-21ssh23:09:061
2394207995624,28cyclictest23-21ksoftirqd/100:23:181
239420799562,20cyclictest2436122-21munin-run22:30:011
239420799562,14cyclictest2473735-21ssh23:54:001
239420799562,12cyclictest2421398-21ssh21:56:431
239420799561,9cyclictest2414263-21ssh21:40:381
2394207995612,12cyclictest23-21ksoftirqd/123:18:461
239420199562,49cyclictest2394396-21munin-run19:15:010
239420199561,51cyclictest2397481-21sshd20:11:230
239421599558,9cyclictest361rcuc/300:34:263
2394215995517,9cyclictest37-21ksoftirqd/300:05:163
2394215995512,10cyclictest37-21ksoftirqd/321:11:283
2394211995547,5cyclictest2436153-21kworker/2:322:46:522
2394211995546,5cyclictest2436153-21kworker/2:323:03:282
2394211995546,5cyclictest2386092-21kworker/2:222:10:012
2394211995546,5cyclictest2386092-21kworker/2:222:05:012
2394211995546,5cyclictest2386092-21kworker/2:222:05:002
2394211995545,7cyclictest2451773-21kworker/2:200:00:392
239420799553,48cyclictest2396214-21cron19:50:001
239420799552,9cyclictest2477266-21ssh00:01:401
2394207995526,26cyclictest23-21ksoftirqd/123:46:111
2394207995523,28cyclictest2395672-21sshd19:38:231
2394207995519,32cyclictest2395214-21latency_hist19:30:011
2394207995516,34cyclictest2428402-21rm22:12:281
239420799551,13cyclictest2400559-21sed21:10:021
239420199553,47cyclictest364-21snmpd19:34:290
239420199552,48cyclictest364-21snmpd20:51:050
239420199552,33cyclictest2396475-21latency_hist19:55:020
239420199552,33cyclictest2396475-21latency_hist19:55:010
239420199551,4cyclictest2395153-21sshd19:25:240
2394215995414,10cyclictest37-21ksoftirqd/321:55:013
2394215995414,10cyclictest37-21ksoftirqd/321:55:013
2394215995413,9cyclictest37-21ksoftirqd/323:52:473
2394211995445,5cyclictest2436153-21kworker/2:323:09:122
2394211995444,6cyclictest2397537-21kworker/2:021:14:522
239420799544,14cyclictest2399857-21modprobe21:00:011
239420799544,14cyclictest2399857-21modprobe21:00:001
239420799544,10cyclictest2398549-21modprobe20:35:011
239420799542,7cyclictest364-21snmpd00:31:111
239420799542,14cyclictest3467602-21systemd-journal23:35:011
239420799542,14cyclictest3467602-21systemd-journal23:35:011
239420799542,11cyclictest2394817-21latency_hist19:20:021
2394207995420,29cyclictest3467602-21systemd-journal20:17:001
2394207995417,34cyclictest2411815-21sed21:35:031
2394207995417,34cyclictest2396553-21latency_hist19:55:021
2394207995417,34cyclictest2396553-21latency_hist19:55:011
2394207995415,35cyclictest2420255-1kworker/u9:122:57:011
2394207995413,35cyclictest0-21swapper/120:29:591
239420799541,12cyclictest309-21in:imuxsock20:00:011
239420799541,11cyclictest2441272-21ssh22:41:101
239420799541,11cyclictest2431547-21ssh22:19:521
239420199544,41cyclictest2394202-21latency_hist19:10:010
239420199543,46cyclictest364-21snmpd21:01:170
2394215995311,8cyclictest37-21ksoftirqd/323:23:513
2394211995345,5cyclictest2451773-21kworker/2:223:50:132
2394211995345,5cyclictest2436153-21kworker/2:323:10:452
2394211995345,5cyclictest2429736-21kworker/2:122:43:402
2394211995344,5cyclictest2386092-21kworker/2:221:40:302
2394211995343,5cyclictest2451773-21kworker/2:223:43:292
2394211995342,7cyclictest2458506-21kworker/2:100:16:412
2394211995339,11cyclictest2386092-21kworker/2:221:50:192
2394211995339,11cyclictest2386092-21kworker/2:221:50:192
239420799533,8cyclictest2397521-21true20:15:001
239420799533,6cyclictest2405638-21ssh21:21:151
239420799533,6cyclictest2405638-21ssh21:21:141
239420799532,9cyclictest2439237-21ssh22:36:381
239420799532,23cyclictest2398976-21grep20:40:011
239420799532,14cyclictest2484671-21sh00:18:231
239420799532,13cyclictest2424721-21sh22:04:301
239420799532,13cyclictest2424721-21sh22:04:301
239420799531,15cyclictest2425333-21sshd22:05:231
239420799531,10cyclictest2474701-21ssh23:55:541
2394207995310,38cyclictest0-21swapper/120:22:391
2394215995213,9cyclictest37-21ksoftirqd/321:28:253
2394215995213,8cyclictest37-21ksoftirqd/300:37:093
239421599520,34cyclictest0-21swapper/321:20:023
2394211995244,5cyclictest2386092-21kworker/2:222:18:242
2394211995243,5cyclictest2451773-21kworker/2:200:22:532
2394211995242,7cyclictest2386092-21kworker/2:220:35:012
239420199521,47cyclictest2399052-21sshd20:40:240
2394215995137,10cyclictest1-21systemd21:45:013
239421599511,14cyclictest2489328-21ssh00:28:523
2394215995110,8cyclictest37-21ksoftirqd/322:05:593
2394211995143,5cyclictest2451773-21kworker/2:200:29:262
2394211995143,5cyclictest2386092-21kworker/2:221:32:562
2394211995142,6cyclictest2386092-21kworker/2:221:26:112
2394211995141,6cyclictest2451773-21kworker/2:223:28:462
239421599508,9cyclictest37-21ksoftirqd/300:17:243
2394215995031,15cyclictest3467602-21systemd-journal20:55:013
2394215995015,8cyclictest37-21ksoftirqd/322:50:493
2394215995011,7cyclictest37-21ksoftirqd/321:34:273
2394215995010,8cyclictest37-21ksoftirqd/323:16:293
2394211995042,5cyclictest2429736-21kworker/2:122:37:022
2394211995042,5cyclictest2386092-21kworker/2:221:55:432
2394211995042,5cyclictest2386092-21kworker/2:221:23:132
2394211995042,5cyclictest2386092-21kworker/2:221:23:132
2394211995042,5cyclictest2386092-21kworker/2:220:56:242
2394211995042,5cyclictest2386092-21kworker/2:220:56:232
2394211995042,5cyclictest2386092-21kworker/2:220:00:012
2394211995041,6cyclictest2386092-21kworker/2:219:44:222
2394211995041,5cyclictest2386092-21kworker/2:222:27:572
2394211995040,6cyclictest2386092-21kworker/2:220:10:012
239421599498,7cyclictest37-21ksoftirqd/323:30:573
239421599498,7cyclictest37-21ksoftirqd/323:30:563
2394215994911,8cyclictest0-21swapper/323:11:163
2394215994910,8cyclictest37-21ksoftirqd/323:04:443
2394211994941,5cyclictest2451773-21kworker/2:223:45:322
2394211994941,5cyclictest2451773-21kworker/2:200:08:422
2394211994941,5cyclictest2386092-21kworker/2:221:45:282
2394211994941,5cyclictest2386092-21kworker/2:220:25:012
2394211994941,5cyclictest2386092-21kworker/2:220:20:012
2394211994941,5cyclictest2386092-21kworker/2:219:26:152
2394211994931,13cyclictest3467602-21systemd-journal19:10:032
239420799492,7cyclictest2399592-21munin-run20:55:001
2394215994810,9cyclictest37-21ksoftirqd/322:10:323
2394211994841,4cyclictest2451773-21kworker/2:223:22:092
2394211994840,5cyclictest2386092-21kworker/2:221:38:532
2394211994839,5cyclictest2386092-21kworker/2:221:10:002
239420799481,12cyclictest364-21snmpd19:30:101
239421599478,9cyclictest37-21ksoftirqd/323:07:023
239421599478,7cyclictest37-21ksoftirqd/321:39:333
239421599474,13cyclictest2400376-21diskmemload22:22:263
2394215994736,7cyclictest2395971-21munin-plugin-st19:45:013
2394211994740,5cyclictest2451773-21kworker/2:223:59:522
2394211994739,5cyclictest2386092-21kworker/2:222:12:272
2394211994739,5cyclictest2386092-21kworker/2:220:15:022
2394211994738,5cyclictest2386092-21kworker/2:219:25:012
239420799471,13cyclictest2397364-21latency_hist20:10:031
2394215994629,13cyclictest2471945-21latency_hist23:50:023
2394215994626,15cyclictest275-21dbus-daemon20:00:003
2394211994638,5cyclictest2386092-21kworker/2:220:50:022
2394211994638,5cyclictest2386092-21kworker/2:219:35:242
239421599458,7cyclictest37-21ksoftirqd/323:29:443
239421599457,9cyclictest37-21ksoftirqd/322:28:393
2394215994512,8cyclictest0-21swapper/322:32:503
2394215994511,8cyclictest0-21swapper/300:14:343
2394211994537,5cyclictest2386092-21kworker/2:220:00:232
2394215994412,8cyclictest0-21swapper/319:15:013
2394215994412,7cyclictest37-21ksoftirqd/322:59:083
2394211994437,4cyclictest2386092-21kworker/2:219:19:322
239420799442,8cyclictest2394246-21grep19:10:021
239421599436,6cyclictest37-21ksoftirqd/322:35:043
2394215994311,7cyclictest0-21swapper/322:20:003
239421599428,8cyclictest0-21swapper/322:45:243
239421599423,4cyclictest131rcu_preempt00:03:313
2394211994234,5cyclictest2386092-21kworker/2:221:04:252
2394211994234,4cyclictest2386092-21kworker/2:219:47:542
2394211994233,6cyclictest2386092-21kworker/2:220:55:012
2394215994128,8cyclictest309-21in:imuxsock19:32:233
2394215994110,5cyclictest0-21swapper/323:36:063
239421599406,6cyclictest37-21ksoftirqd/300:23:593
2394215994029,8cyclictest3467602-21systemd-journal19:47:013
2394211994032,5cyclictest2386092-21kworker/2:220:27:272
2394211994025,11cyclictest1-21systemd19:35:002
239421599396,7cyclictest37-21ksoftirqd/321:47:163
2394215993926,8cyclictest2399848-21sendmail21:00:013
2394215993926,8cyclictest2399848-21sendmail21:00:003
2394215993926,11cyclictest37-21ksoftirqd/320:15:003
2394215993924,11cyclictest2397740-21cron20:17:003
2394211993931,5cyclictest2386092-21kworker/2:220:37:232
239421599388,6cyclictest0-21swapper/322:04:143
239421599388,6cyclictest0-21swapper/322:04:133
239421599385,7cyclictest37-21ksoftirqd/320:04:233
239421599381,12cyclictest2399247-21latency_hist20:45:033
239421599374,8cyclictest37-21ksoftirqd/319:51:213
239421599374,8cyclictest37-21ksoftirqd/319:51:213
239421599362,8cyclictest37-21ksoftirqd/320:30:243
239421599362,5cyclictest131rcu_preempt21:10:003
2394211993628,5cyclictest2386092-21kworker/2:219:52:132
2394211993628,5cyclictest2386092-21kworker/2:219:52:122
239421599358,22cyclictest2400211-21latency_hist21:05:023
2394215993526,7cyclictest37-21ksoftirqd/319:20:023
2394215993524,9cyclictest37-21ksoftirqd/319:40:013
2394215993422,9cyclictest275-21dbus-daemon20:10:013
2394215993420,10cyclictest309-21in:imuxsock20:38:233
2394215993410,19cyclictest2399476-21latency_hist20:50:023
2394211993322,7cyclictest2399068-21sshd20:43:222
239421599322,5cyclictest2394951-21munin-run19:25:003
2394215993224,5cyclictest2395176-21cron19:30:013
2394215993123,5cyclictest2398244-21latency_hist20:25:013
2394215993118,10cyclictest2398288-21sshd20:29:243
239421599307,19cyclictest2394279-21latency_hist19:10:013
2394211992818,7cyclictest30-21ksoftirqd/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional