You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-04 - 12:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Sep 04, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3561554998415,34cyclictest0-21swapper/019:55:020
3561557997728,45cyclictest3619989-21sh23:00:581
356155799764,68cyclictest3598528-21ssh22:15:021
3561554997628,43cyclictest3640736-21ssh23:45:200
3561554997621,38cyclictest3625427-21ssh23:12:560
3561554997511,31cyclictest3615536-21kworker/u8:323:43:110
3561554997511,31cyclictest3615536-21kworker/u8:323:43:110
3561554997330,38cyclictest3568028-21ssh21:10:090
3561554997330,38cyclictest3568028-21ssh21:10:080
3561554997230,15cyclictest3579495-21munin-run21:35:000
3561554997230,15cyclictest3579495-21munin-run21:35:000
3561554997228,40cyclictest3608937-21sshd22:37:250
3561557997163,4cyclictest0-21swapper/122:45:021
3561554997124,42cyclictest3648893-21sh00:03:020
3561554997124,14cyclictest3662068-21ssh00:30:580
3561554997124,14cyclictest3662068-21ssh00:30:570
3561562997011,54cyclictest3649730-21ssh00:05:012
3561562997010,55cyclictest3631002-21cron23:25:012
3561562997010,55cyclictest3631002-21cron23:25:002
3561554997019,46cyclictest3570425-21ssh21:15:130
3561557996926,39cyclictest3628846-21latency_hist23:20:021
356155799681,37cyclictest3646407-21ssh23:57:441
3561554996821,42cyclictest3636959-21ssh23:37:360
356155799674,57cyclictest3584292-21grep21:45:021
3561554996729,4cyclictest3666098-21munin-run00:40:000
3561554996728,35cyclictest3660777-21ssh00:28:340
3561554996726,37cyclictest3467602-21systemd-journal23:00:000
3561554996725,37cyclictest3580270-21ssh21:36:170
3561554996723,38cyclictest3594028-21ssh22:05:200
3561554996720,41cyclictest3572845-21ssh21:20:140
3561554996720,41cyclictest3572845-21ssh21:20:130
356156299661,16cyclictest3571838-21ssh21:18:282
3561557996627,35cyclictest3664237-21sh00:35:381
3561554996627,34cyclictest16850irq/134-800000022:20:020
3561554996626,10cyclictest3582274-21sshd21:40:240
3561554996624,37cyclictest3566428-21cron20:45:000
3561562996531,30cyclictest3628644-21cron23:19:592
3561557996529,33cyclictest3644243-21sh23:53:101
3561554996522,39cyclictest0-21swapper/023:59:000
3561554996522,38cyclictest0-21swapper/021:10:010
3561557996436,24cyclictest3467602-21systemd-journal19:38:241
3561557996436,24cyclictest3467602-21systemd-journal19:38:241
3561554996429,4cyclictest3585071-21sshd21:46:240
3561554996425,34cyclictest3630487-21ssh23:23:440
3561554996425,34cyclictest3630487-21ssh23:23:430
356156299634,52cyclictest284-21systemd-logind19:35:012
3561562996325,34cyclictest3567730-21diskmemload00:08:202
3561562996314,44cyclictest3621610-21cron23:05:002
3561557996333,26cyclictest3634963-21sshd23:33:251
3561557996333,26cyclictest3634963-21sshd23:33:241
3561557996331,29cyclictest3565659-21munin-plugin-st20:30:001
3561554996323,36cyclictest3642686-21kworker/u8:400:10:000
3561554996323,11cyclictest3656172-21ssh00:18:470
3561554996322,36cyclictest0-21swapper/000:15:020
3561554996321,10cyclictest3593146-21ssh22:03:470
3561562996231,27cyclictest3467602-21systemd-journal00:32:242
3561562996231,27cyclictest3467602-21systemd-journal00:32:242
356156299622,55cyclictest3570121-21ssh21:15:012
356156299622,55cyclictest3570121-21ssh21:15:012
3561562996224,35cyclictest3603108-21wc22:25:012
3561562996224,34cyclictest3585120-21sh21:46:302
3561557996225,34cyclictest3611507-21sh22:43:191
3561557996224,34cyclictest364-21snmpd20:21:371
356155799621,56cyclictest3648301-21sh00:01:321
356155799621,56cyclictest3609534-21sh22:39:101
3561570996139,17cyclictest1-21systemd23:39:593
356156299612,53cyclictest3588915-21sh21:55:002
356156299611,18cyclictest3603291-21sh22:25:062
356156299611,14cyclictest1-21systemd22:50:002
3561557996130,27cyclictest275-21dbus-daemon23:30:001
3561557996123,34cyclictest3606348-21sh22:31:501
3561554996126,31cyclictest3614628-21latency_hist22:50:030
3561554996124,33cyclictest120-21kswapd022:35:010
3561554996122,35cyclictest3644077-21ssh23:52:480
3561554996119,37cyclictest3610904-21ssh22:41:500
356155799602,53cyclictest3587742-21ssh21:52:121
3561557996023,34cyclictest3650057-21sh00:05:191
3561557996020,35cyclictest3652462-21sh00:10:251
3561557996020,35cyclictest23-21ksoftirqd/123:46:471
3561554996023,32cyclictest3591269-21cron22:00:000
356156299592,53cyclictest3597203-21ssh22:12:232
356156299591,54cyclictest3638881-21ssh23:41:362
356156299591,54cyclictest3638881-21ssh23:41:362
356156299591,54cyclictest3624811-21ssh23:11:272
356156299591,13cyclictest3659869-21ssh00:26:262
356155799593,51cyclictest3584885-21sh21:46:001
356155799593,50cyclictest3564352-21cron20:05:011
3561557995919,35cyclictest3272098-1kworker/u9:221:39:071
3561554995924,7cyclictest0-21swapper/023:18:360
3561554995920,29cyclictest16850irq/134-800000022:25:590
3561554995920,10cyclictest3576559-21rm21:28:450
356157099581,10cyclictest3666090-21sh00:40:003
356156299584,22cyclictest131rcu_preempt00:35:342
356156299583,50cyclictest3589747-21sh21:56:252
356156299582,52cyclictest3658574-21ssh00:23:532
356156299582,52cyclictest3658574-21ssh00:23:532
356156299581,53cyclictest3615374-21ssh22:51:162
356156299581,52cyclictest364-21snmpd19:57:272
356156299581,14cyclictest3653626-21ssh00:13:132
356156299581,11cyclictest3611851-21sh22:44:012
356156299581,10cyclictest3606112-21ssh22:31:142
3561562995810,43cyclictest3467602-21systemd-journal00:19:592
356155799582,51cyclictest3625605-21ssh23:13:171
356155799582,50cyclictest3618442-21sh22:58:001
356155799581,53cyclictest3662755-21sh00:32:361
356155799581,53cyclictest3662755-21sh00:32:361
356155799581,53cyclictest3575204-21sh21:25:181
356155799581,52cyclictest3615929-21sh22:52:381
356155799581,52cyclictest3578341-21ssh21:32:221
356155799581,52cyclictest3578341-21ssh21:32:221
3561554995826,28cyclictest3566941-21latency_hist20:55:020
3561554995826,28cyclictest3566941-21latency_hist20:55:020
3561554995822,4cyclictest3563015-21sshd19:35:240
3561554995822,4cyclictest3563015-21sshd19:35:240
3561554995818,36cyclictest0-21swapper/020:35:020
3561554995818,36cyclictest0-21swapper/020:35:020
3561554995818,10cyclictest3623991-21munin-run23:10:000
356156299573,48cyclictest3647383-40mandb00:00:012
356156299573,48cyclictest3567487-21systemd-run21:05:002
356156299572,53cyclictest3580755-21ssh21:37:252
356156299572,51cyclictest3567202-21munin-run21:00:012
356156299572,50cyclictest3610127-21latency_hist22:40:032
356156299572,49cyclictest3599511-21ssh22:17:142
356156299572,49cyclictest3577263-21latency_hist21:30:022
356156299571,52cyclictest3594326-21ssh22:06:052
356156299571,52cyclictest3564365-21munin-run20:05:002
3561557995730,23cyclictest3569605-21sh21:13:481
3561557995730,23cyclictest3569605-21sh21:13:471
356155799572,51cyclictest3657451-21ssh00:21:111
356155799572,51cyclictest3657451-21ssh00:21:101
356155799572,51cyclictest3636907-21sh23:37:291
356155799572,50cyclictest3597053-21ssh22:12:041
3561557995724,30cyclictest3654701-21sh00:15:151
3561557995721,33cyclictest3567730-21diskmemload23:20:241
3561557995721,33cyclictest3567730-21diskmemload23:20:231
356155799571,52cyclictest3570684-21ssh21:15:461
3561554995726,15cyclictest3615159-21ssh22:50:480
356155499572,19cyclictest1-21systemd22:15:000
3561554995721,28cyclictest16850irq/134-800000023:30:190
3561554995721,28cyclictest16850irq/134-800000023:30:180
3561570995635,16cyclictest3467602-21systemd-journal19:44:593
356156299562,50cyclictest364-21snmpd23:27:022
356156299562,49cyclictest3574695-21sh21:24:422
356156299562,49cyclictest3574695-21sh21:24:422
3561562995620,33cyclictest3567730-21diskmemload22:57:182
3561562995615,35cyclictest275-21dbus-daemon23:35:002
3561562995615,35cyclictest275-21dbus-daemon23:35:002
356156299561,51cyclictest3566945-21munin-run20:55:002
356156299561,51cyclictest3566945-21munin-run20:55:002
356156299561,50cyclictest364-21snmpd19:47:572
356156299561,50cyclictest3583669-21ssh21:43:392
356156299561,19cyclictest3622948-21sh23:07:272
356156299561,12cyclictest3567730-21diskmemload23:45:192
356155799562,50cyclictest3605143-21ssh22:29:301
356155799561,51cyclictest3638378-21ssh23:40:241
356155799561,51cyclictest3638378-21ssh23:40:231
356155799561,51cyclictest3593958-21ssh22:05:101
356155799561,51cyclictest3574724-21ssh21:24:461
356155799561,51cyclictest3574724-21ssh21:24:451
356155799561,50cyclictest3564302-21cat20:00:021
3561554995620,32cyclictest0-21swapper/000:22:180
3561554995620,32cyclictest0-21swapper/000:22:170
3561554995614,34cyclictest16850irq/134-800000023:29:410
3561557995525,26cyclictest3566480-21date20:45:021
356155799552,47cyclictest3564802-21cut20:10:021
356155799551,50cyclictest3659441-21ssh00:25:211
356155799551,50cyclictest3591925-21ssh22:01:011
356155799551,50cyclictest3590916-21ssh21:59:171
356155799551,50cyclictest3567152-21latency_hist20:55:031
356155799551,50cyclictest1-21systemd19:25:011
356155799551,49cyclictest3565349-21latency_hist20:20:031
356155799551,49cyclictest1-21systemd19:30:021
356155799551,49cyclictest1-21systemd19:30:011
3561554995530,21cyclictest3564517-21latency_hist20:05:020
3561554995527,13cyclictest3602896-21ssh22:24:450
3561554995522,31cyclictest0-21swapper/023:05:020
3561554995522,28cyclictest3565480-21latency_hist20:25:020
3561570995420,8cyclictest131rcu_preempt23:26:273
356156299545,44cyclictest3567713-21cron21:10:002
3561562995446,5cyclictest3552422-21kworker/2:019:40:002
3561562995446,5cyclictest3552422-21kworker/2:019:40:002
356156299542,48cyclictest364-21snmpd20:34:132
356156299542,48cyclictest364-21snmpd20:34:132
356156299541,11cyclictest3645112-21latency_hist23:55:012
356155799542,46cyclictest3564853-21cron20:14:591
356155799541,49cyclictest3622651-21ssh23:06:511
3561554995430,20cyclictest3567608-21latency_hist21:05:020
3561554995428,22cyclictest3588008-21sh21:52:470
3561554995420,3cyclictest3567160-21sshd20:55:250
3561570995335,12cyclictest3467602-21systemd-journal23:10:013
3561570995334,14cyclictest275-21dbus-daemon23:25:003
3561570995334,14cyclictest275-21dbus-daemon23:25:003
356156299533,44cyclictest3593853-21latency_hist22:05:032
356156299532,47cyclictest3565409-21munin-run20:25:012
356156299531,13cyclictest3577878-21ssh21:31:132
356156299531,13cyclictest3577878-21ssh21:31:132
356156299531,12cyclictest364-21snmpd20:13:282
356155799531,50cyclictest3601834-21ssh22:22:061
356155799531,48cyclictest3566381-21latency_hist20:40:031
356155799531,48cyclictest3561808-21latency_hist19:15:011
3561554995329,18cyclictest3562064-21cat19:20:010
3561554995325,22cyclictest3467602-21systemd-journal20:38:250
356156299522,46cyclictest3637284-21ssh23:38:182
356156299521,46cyclictest3565083-21modprobe20:15:242
356155799527,41cyclictest275-21dbus-daemon19:20:021
3561557995227,21cyclictest3567198-21cat20:59:591
3561557995220,29cyclictest3567698-21sshd21:06:241
356157099516,29cyclictest361rcuc/321:05:023
3561570995133,13cyclictest309-21in:imuxsock19:30:023
356157099512,14cyclictest3605369-21latency_hist22:30:023
3561562995137,10cyclictest1-21systemd19:20:002
356156299513,44cyclictest3562548-21munin-run19:30:012
356156299513,44cyclictest3562548-21munin-run19:30:012
3561562995116,30cyclictest3560720-21kworker/u8:320:10:012
356155799511,45cyclictest3565956-21date20:35:031
356155799511,45cyclictest3565956-21date20:35:021
356155799511,45cyclictest1-21systemd20:55:011
3561554995129,4cyclictest3467602-21systemd-journal20:18:310
3561570995033,12cyclictest0-21swapper/323:35:023
3561570995033,12cyclictest0-21swapper/323:35:013
356156299502,43cyclictest364-21snmpd19:10:262
356155799502,41cyclictest3467602-21systemd-journal20:50:001
3561554995030,16cyclictest3467602-21systemd-journal19:59:590
3561554995027,18cyclictest3563481-21latency_hist19:45:020
356157099496,11cyclictest37-21ksoftirqd/322:46:423
356156299495,40cyclictest3565635-21sshd20:27:402
356156299492,41cyclictest311-21rs:main2
356156299491,5cyclictest3566236-21cat20:40:022
356156299491,44cyclictest3563833-21munin-run19:55:002
356156299491,44cyclictest3562279-21sshd19:24:252
356156299491,43cyclictest3563302-21sshd19:42:242
356155799492,42cyclictest3562760-21modprobe19:30:021
356155799491,43cyclictest3563298-21sendmail-mta19:41:061
3561554994934,10cyclictest3564867-21munin-plugin-st20:15:000
356156299482,43cyclictest3561550-21cyclictest19:10:012
356155799482,43cyclictest0-21swapper/119:10:021
3561554994831,13cyclictest3467602-21systemd-journal19:27:310
3561554994831,13cyclictest3467602-21systemd-journal19:27:300
3561554994831,11cyclictest3562787-21cron19:34:590
3561554994828,15cyclictest275-21dbus-daemon20:50:000
356155499481,10cyclictest1-21systemd20:10:010
3561570994735,7cyclictest37-21ksoftirqd/300:10:033
3561570994732,9cyclictest1-21systemd22:20:013
3561570994732,11cyclictest1-21systemd23:00:013
3561570994732,10cyclictest275-21dbus-daemon22:00:003
3561570994731,12cyclictest3570092-21cron21:15:003
3561570994731,12cyclictest3570092-21cron21:15:003
3561570994726,16cyclictest3660537-21sshd00:28:013
3561570994710,6cyclictest131rcu_preempt00:00:003
3561557994730,13cyclictest1-21systemd19:50:001
356155799471,43cyclictest3567448-21sshd21:02:241
356155799471,42cyclictest0-21swapper/119:50:421
3561554994732,10cyclictest3563542-21modprobe19:45:020
3561554994710,33cyclictest3565674-21unin-run20:30:020
3561570994612,4cyclictest3589016-21latency_hist21:55:023
356156299462,41cyclictest3566425-21sshd20:44:242
3561554994628,13cyclictest3561751-21cron19:15:010
3561570994537,6cyclictest37-21ksoftirqd/322:30:573
3561570994531,10cyclictest275-21dbus-daemon21:45:023
3561554994527,13cyclictest3561659-21sed19:10:020
3561570994434,8cyclictest37-21ksoftirqd/320:29:153
3561570994428,13cyclictest37-21ksoftirqd/323:45:023
3561570994428,12cyclictest275-21dbus-daemon19:20:003
3561570994426,13cyclictest3467602-21systemd-journal20:39:243
3561570994414,24cyclictest361rcuc/319:50:023
3561570994331,9cyclictest37-21ksoftirqd/321:37:063
3561570994319,7cyclictest361rcuc/323:51:593
3561554994330,10cyclictest3562272-21sshd19:22:290
356157099427,5cyclictest131rcu_preempt00:33:463
356157099427,5cyclictest131rcu_preempt00:33:463
3561570994231,9cyclictest37-21ksoftirqd/300:11:263
3561570994231,8cyclictest3566668-21sshd20:47:443
3561570994228,11cyclictest37-21ksoftirqd/322:00:163
3561570994228,10cyclictest3467602-21systemd-journal21:09:593
3561570994227,12cyclictest37-21ksoftirqd/322:54:413
3561570994212,6cyclictest131rcu_preempt22:13:343
356157099421,10cyclictest0-21swapper/321:34:423
356157099421,10cyclictest0-21swapper/321:34:413
356157099416,9cyclictest3648333-21ssh00:01:453
3561570994131,7cyclictest37-21ksoftirqd/319:39:003
3561570994131,7cyclictest37-21ksoftirqd/319:38:593
3561570994128,10cyclictest37-21ksoftirqd/320:35:033
3561570994128,10cyclictest37-21ksoftirqd/320:35:023
3561570994127,11cyclictest37-21ksoftirqd/323:17:013
3561570994126,11cyclictest37-21ksoftirqd/323:12:353
3561570994118,9cyclictest0-21swapper/323:44:163
3561570994118,9cyclictest0-21swapper/323:44:153
3561570994032,6cyclictest37-21ksoftirqd/322:05:493
3561570994029,8cyclictest37-21ksoftirqd/320:17:213
3561570994027,10cyclictest1-21systemd22:25:013
356157099402,6cyclictest131rcu_preempt19:59:593
3561570994026,11cyclictest37-21ksoftirqd/300:15:263
3561570994026,11cyclictest3562584-21sed19:30:033
3561570994026,11cyclictest3562584-21sed19:30:023
3561570994024,12cyclictest3467602-21systemd-journal22:35:253
3561570994023,13cyclictest3575018-21sed21:25:033
3561570994023,13cyclictest3575018-21sed21:25:023
3561570994010,6cyclictest131rcu_preempt21:27:033
356157099399,6cyclictest131rcu_preempt22:45:023
3561570993929,7cyclictest3563831-21munin-plugin-st19:55:003
3561570993827,8cyclictest37-21ksoftirqd/323:02:153
3561570993825,9cyclictest37-21ksoftirqd/300:20:303
3561570993825,9cyclictest37-21ksoftirqd/300:20:303
3561570993819,11cyclictest37-21ksoftirqd/321:16:443
3561570993730,4cyclictest3565396-21cron20:25:003
3561570993722,10cyclictest37-21ksoftirqd/320:10:023
3561570993713,6cyclictest131rcu_preempt21:45:503
3561570993628,5cyclictest3567195-21sendmail21:00:003
3561570993627,6cyclictest37-21ksoftirqd/320:52:013
3561570993627,6cyclictest37-21ksoftirqd/320:52:013
3561570993627,6cyclictest3564614-21munin-plugin-st20:10:003
3561570993525,6cyclictest3562291-21sh19:25:003
3561570993522,10cyclictest37-21ksoftirqd/319:15:013
3561570993325,6cyclictest37-21ksoftirqd/320:00:483
3561570993318,11cyclictest37-21ksoftirqd/320:44:393
3561570993018,9cyclictest3561582-21cut19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional