You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-04 - 18:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Sep 04, 2026 12:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3949716998926,14cyclictest3967980-21ssh09:35:390
3949716998828,11cyclictest3994679-21ssh10:33:320
3949716998526,13cyclictest4006014-21ssh10:57:500
3949716998418,12cyclictest4016050-21ssh11:19:130
3949716998418,12cyclictest4016050-21ssh11:19:130
3949716998223,12cyclictest4019094-21ssh11:25:190
3949716998223,12cyclictest4019094-21ssh11:25:190
3949716998123,14cyclictest4024629-21ssh11:37:350
3949716998021,14cyclictest4036406-21ssh12:02:330
3949716997928,15cyclictest4026189-21ssh11:40:430
3949716997838,36cyclictest16850irq/134-800000010:46:240
3949716997727,18cyclictest4043155-21ssh12:17:260
3949716997727,11cyclictest3984472-21ssh10:11:130
3949716997725,15cyclictest3998032-21ssh10:40:160
3949716997725,15cyclictest3998032-21ssh10:40:160
3949716997722,12cyclictest3974145-21ssh09:49:190
3949716997722,12cyclictest3974145-21ssh09:49:190
394971699762,41cyclictest3956851-21ssh09:11:470
3949716997621,11cyclictest4003895-21ssh10:53:140
3949716997621,11cyclictest4003895-21ssh10:53:130
394971699761,14cyclictest3955897-21sh09:10:010
3949716997440,30cyclictest16850irq/134-800000011:15:010
3949716997329,39cyclictest48-21kcompactd012:35:120
3949716997328,15cyclictest3989051-21ssh10:21:080
3949716997320,12cyclictest4033120-21ssh11:55:310
3949716997314,14cyclictest3970901-21ssh09:42:020
3949716997252,16cyclictest12-21ksoftirqd/010:00:020
3949716997229,25cyclictest3987405-21ssh10:17:510
3949716997229,25cyclictest3987405-21ssh10:17:500
3949716997225,13cyclictest3959166-21ssh09:16:520
3949716997223,12cyclictest3996586-21ssh10:37:250
3949716997167,2cyclictest0-21swapper/007:40:020
3949716997128,11cyclictest3967066-21ssh09:34:030
394971699712,6cyclictest4046615-21sed12:25:010
3949716997124,14cyclictest4046794-21ssh12:25:100
3949716997124,14cyclictest4046794-21ssh12:25:100
3949716997027,16cyclictest4040531-21ssh12:11:310
3949716996924,10cyclictest12-21ksoftirqd/007:20:010
394971699691,63cyclictest0-21swapper/012:35:010
394971699691,63cyclictest0-21swapper/008:20:000
3949716996827,9cyclictest4031211-21ssh11:51:260
3949716996827,15cyclictest4021539-21ssh11:30:340
3949716996826,14cyclictest4029852-21ssh11:48:550
3949723996641,8cyclictest3955932-21kworker/2:009:40:032
3949716996625,12cyclictest4010094-21ssh11:06:150
3949716996618,43cyclictest16850irq/134-800000011:01:330
3949716996615,12cyclictest3983361-21ssh10:09:120
3949716996529,31cyclictest12-21ksoftirqd/010:30:020
3949716996425,4cyclictest0-21swapper/009:28:430
3949716996316,10cyclictest3978153-21ssh09:57:440
3949716996224,11cyclictest12-21ksoftirqd/008:10:010
3949716996224,11cyclictest12-21ksoftirqd/008:10:010
394971699621,26cyclictest0-21swapper/009:23:360
394971699611,4cyclictest3975683-21sshd09:52:240
3949723996037,14cyclictest3953823-21kworker/2:209:21:482
3949716996014,19cyclictest16850irq/134-800000012:09:180
3949716996014,19cyclictest16850irq/134-800000012:09:180
3949723995950,5cyclictest3988394-21kworker/2:311:05:022
3949723995949,6cyclictest4016433-21kworker/2:111:40:002
394971699591,27cyclictest0-21swapper/011:22:170
3949723995749,5cyclictest4016433-21kworker/2:111:45:082
3949723995739,12cyclictest3467602-21systemd-journal09:20:002
3949723995647,6cyclictest4016433-21kworker/2:112:25:392
3949723995647,6cyclictest4016433-21kworker/2:112:25:382
3949723995647,5cyclictest4016433-21kworker/2:112:21:242
394971699561,10cyclictest1-21systemd07:27:240
3949723995547,5cyclictest3988394-21kworker/2:310:49:502
3949723995547,5cyclictest3955932-21kworker/2:009:53:452
3949723995546,6cyclictest3955932-21kworker/2:009:58:232
3949723995546,5cyclictest3988394-21kworker/2:310:22:072
3949723995545,5cyclictest4016433-21kworker/2:111:41:402
3949716995520,14cyclictest1-21systemd07:15:000
394971699551,28cyclictest0-21swapper/007:41:120
3949726995431,18cyclictest4023533-21diskmemload11:35:023
3949723995446,5cyclictest4016433-21kworker/2:112:15:022
3949723995445,6cyclictest3955932-21kworker/2:009:30:332
3949723995444,6cyclictest3953823-21kworker/2:209:10:012
3949723995436,15cyclictest1-21systemd11:10:002
3949723995346,5cyclictest4016433-21kworker/2:112:16:182
3949723995345,5cyclictest3988394-21kworker/2:311:33:162
3949723995345,5cyclictest3953823-21kworker/2:209:13:272
3949723995343,6cyclictest3988394-21kworker/2:310:45:012
3949723995343,6cyclictest3988394-21kworker/2:310:45:012
3949716995334,14cyclictest3467602-21systemd-journal08:05:010
394971699533,32cyclictest3950933-21kworker/u8:108:25:020
3949726995220,23cyclictest37-21ksoftirqd/311:49:483
3949723995244,5cyclictest4023416-21kworker/2:012:31:442
3949723995244,5cyclictest3988394-21kworker/2:310:36:592
3949723995244,5cyclictest3988394-21kworker/2:310:33:462
3949723995244,5cyclictest3941960-21kworker/2:108:10:022
3949723995244,5cyclictest3941960-21kworker/2:108:10:012
3949723995243,5cyclictest3988394-21kworker/2:311:29:402
3949723995243,5cyclictest3988394-21kworker/2:311:29:402
3949723995243,5cyclictest3955932-21kworker/2:010:10:022
3949723995237,7cyclictest3988394-21kworker/2:311:18:082
3949723995237,7cyclictest3988394-21kworker/2:311:18:082
3949716995242,7cyclictest19350irq/34-mmc108:55:020
394971699522,29cyclictest19350irq/34-mmc109:00:030
3949726995136,11cyclictest3467602-21systemd-journal11:15:003
3949726995120,22cyclictest37-21ksoftirqd/311:07:103
3949723995143,5cyclictest4023416-21kworker/2:012:37:522
3949723995142,5cyclictest4016433-21kworker/2:112:05:082
3949723995142,5cyclictest4016433-21kworker/2:112:05:072
3949723995142,5cyclictest3953823-21kworker/2:209:30:012
3949723995141,5cyclictest3955932-21kworker/2:010:11:082
3949716995123,25cyclictest12-21ksoftirqd/007:25:020
3949716995121,4cyclictest3950947-21latency_hist07:35:020
3949726995030,5cyclictest37-21ksoftirqd/310:48:423
3949726995022,21cyclictest3467602-21systemd-journal09:40:003
3949723995042,5cyclictest4016433-21kworker/2:112:03:182
3949723995042,5cyclictest4016433-21kworker/2:111:52:092
3949723995042,4cyclictest3941960-21kworker/2:107:10:022
3949723995041,6cyclictest3953823-21kworker/2:208:45:012
3949723995041,5cyclictest4016433-21kworker/2:111:55:232
3949723995041,5cyclictest3967497-21kworker/2:110:02:072
3949723995036,6cyclictest3955932-21kworker/2:009:48:372
3949723995036,6cyclictest3955932-21kworker/2:009:48:372
3949723995025,8cyclictest3988394-21kworker/2:311:14:592
394972699498,12cyclictest131rcu_preempt12:35:003
3949726994934,7cyclictest37-21ksoftirqd/308:20:023
3949726994916,21cyclictest131rcu_preempt11:26:273
3949726994916,21cyclictest131rcu_preempt11:26:263
3949723994941,5cyclictest4016433-21kworker/2:111:21:502
3949723994941,5cyclictest3988394-21kworker/2:310:54:182
3949723994941,5cyclictest3988394-21kworker/2:310:54:182
3949723994941,5cyclictest3955932-21kworker/2:009:40:002
394971699492,43cyclictest3954338-21basename08:39:590
3949716994918,9cyclictest3467602-21systemd-journal08:45:000
3949726994837,8cyclictest37-21ksoftirqd/310:05:023
3949726994833,12cyclictest3467602-21systemd-journal10:30:033
3949726994832,13cyclictest37-21ksoftirqd/309:00:033
3949726994832,12cyclictest275-21dbus-daemon07:40:013
3949726994813,6cyclictest131rcu_preempt10:40:003
3949723994840,5cyclictest3988394-21kworker/2:310:55:072
3949723994840,5cyclictest3941960-21kworker/2:108:25:002
3949726994734,9cyclictest37-21ksoftirqd/309:48:153
3949726994734,9cyclictest37-21ksoftirqd/309:48:153
3949726994731,10cyclictest309-21in:imuxsock11:55:003
3949726994717,21cyclictest361rcuc/310:18:163
3949726994717,21cyclictest361rcuc/310:18:163
3949723994740,4cyclictest3953823-21kworker/2:208:50:012
3949723994739,5cyclictest3953823-21kworker/2:209:00:012
3949720994730,12cyclictest3467602-21systemd-journal08:30:011
3949720994712,16cyclictest23-21ksoftirqd/112:40:021
3949726994634,8cyclictest4044310-21sendmail-msp12:20:003
3949726994630,12cyclictest3467602-21systemd-journal09:50:043
3949726994624,17cyclictest37-21ksoftirqd/309:15:013
3949726994619,23cyclictest37-21ksoftirqd/312:24:303
394972699461,6cyclictest120-21kswapd009:30:003
3949723994638,5cyclictest3988394-21kworker/2:310:27:202
3949723994637,5cyclictest3941960-21kworker/2:107:15:022
394972399461,29cyclictest3952499-21kworker/u8:008:20:022
3949720994626,18cyclictest23-21ksoftirqd/107:15:021
394971699468,21cyclictest0-21swapper/008:00:010
3949726994520,5cyclictest37-21ksoftirqd/312:40:003
3949723994537,5cyclictest3955932-21kworker/2:010:16:532
3949723994537,5cyclictest3955932-21kworker/2:010:16:532
3949723994531,11cyclictest1-21systemd07:59:592
3949720994536,6cyclictest23-21ksoftirqd/112:11:001
3949720994534,6cyclictest309-21in:imuxsock09:59:591
3949720994531,10cyclictest3467602-21systemd-journal09:00:001
394972699445,32cyclictest4039720-21wc12:10:023
394972699445,32cyclictest4039720-21wc12:10:023
3949723994428,11cyclictest275-21dbus-daemon07:20:012
394972099447,4cyclictest131rcu_preempt10:05:021
3949720994429,11cyclictest23-21ksoftirqd/110:55:021
3949720994429,11cyclictest23-21ksoftirqd/110:55:021
3949726994331,9cyclictest3467602-21systemd-journal11:57:323
3949726994329,10cyclictest3954319-21cat08:40:003
3949726994326,13cyclictest275-21dbus-daemon07:55:013
3949726994326,13cyclictest275-21dbus-daemon07:55:013
3949726994326,12cyclictest37-21ksoftirqd/310:00:023
3949726994326,12cyclictest275-21dbus-daemon07:30:013
3949723994337,4cyclictest3941960-21kworker/2:108:12:552
3949723994335,5cyclictest3941960-21kworker/2:108:05:012
3949723994329,10cyclictest3951729-21cron07:50:002
394972099434,7cyclictest131rcu_preempt10:40:031
3949720994336,5cyclictest23-21ksoftirqd/111:15:021
3949720994332,8cyclictest23-21ksoftirqd/110:43:501
3949720994332,8cyclictest23-21ksoftirqd/110:43:501
3949720994328,11cyclictest3467602-21systemd-journal08:10:031
394972099432,3cyclictest131rcu_preempt11:56:021
3949726994233,7cyclictest37-21ksoftirqd/310:25:023
3949726994230,9cyclictest37-21ksoftirqd/307:20:013
3949726994226,12cyclictest275-21dbus-daemon08:55:013
3949726994220,16cyclictest275-21dbus-daemon09:10:003
3949726994218,5cyclictest37-21ksoftirqd/309:23:563
394972699421,5cyclictest131rcu_preempt11:02:283
394972699421,3cyclictest131rcu_preempt09:17:413
3949726994211,5cyclictest131rcu_preempt07:35:003
3949723994234,5cyclictest3953823-21kworker/2:209:00:032
394972099424,34cyclictest3467602-21systemd-journal09:35:021
3949720994230,8cyclictest3952749-21sshd08:09:241
3949720994230,8cyclictest3952749-21sshd08:09:231
3949720994228,11cyclictest3954574-21sshd08:44:231
3949720994227,11cyclictest3467602-21systemd-journal07:35:001
3949720994226,12cyclictest23-21ksoftirqd/112:17:561
394972099421,3cyclictest131rcu_preempt12:04:111
394972099421,3cyclictest131rcu_preempt09:39:261
3949726994125,9cyclictest309-21in:imuxsock08:49:543
3949726994125,11cyclictest4007037-21systemd-run11:00:003
3949726994124,12cyclictest4016406-21diskmemload11:20:023
3949726994124,12cyclictest4016406-21diskmemload11:20:013
3949723994134,4cyclictest3941960-21kworker/2:107:32:262
3949723994132,5cyclictest3941960-21kworker/2:107:55:002
3949723994132,5cyclictest3941960-21kworker/2:107:55:002
3949723994131,7cyclictest30-21ksoftirqd/208:40:022
3949723994118,18cyclictest0-21swapper/207:45:012
394972099414,3cyclictest131rcu_preempt10:09:151
3949720994132,6cyclictest23-21ksoftirqd/112:05:401
3949720994132,6cyclictest23-21ksoftirqd/112:05:401
3949720994130,6cyclictest3952222-21cron08:00:001
3949720994126,11cyclictest275-21dbus-daemon10:20:011
3949720994126,11cyclictest275-21dbus-daemon10:20:011
394971699411,4cyclictest131rcu_preempt08:50:020
394971699411,28cyclictest0-21swapper/008:14:520
394972699403,3cyclictest131rcu_preempt10:26:263
394972699402,4cyclictest0-21swapper/312:14:533
3949726994021,14cyclictest37-21ksoftirqd/312:02:163
3949726994021,14cyclictest3467602-21systemd-journal08:10:013
3949726994021,14cyclictest3467602-21systemd-journal08:10:013
3949726994011,23cyclictest3967452-21modprobe09:35:003
394972399402,7cyclictest0-21swapper/207:35:202
394972099403,5cyclictest3965095-21munin-run09:30:001
3949720994028,8cyclictest275-21dbus-daemon08:25:011
3949720994028,10cyclictest23-21ksoftirqd/110:30:091
3949720994028,10cyclictest23-21ksoftirqd/109:22:211
3949720994026,10cyclictest23-21ksoftirqd/111:49:051
3949720994025,11cyclictest275-21dbus-daemon12:25:001
3949720994024,12cyclictest1-21systemd10:30:011
394972099402,3cyclictest131rcu_preempt07:25:001
3949720994023,12cyclictest275-21dbus-daemon09:50:011
3949720994023,12cyclictest275-21dbus-daemon09:50:011
394972099401,3cyclictest131rcu_preempt08:50:011
394972099401,20cyclictest0-21swapper/110:11:401
3949716994024,11cyclictest1-21systemd08:55:000
394972699394,25cyclictest4025673-21grep11:40:003
3949726993928,8cyclictest3953536-21sshd08:23:243
3949726993927,8cyclictest37-21ksoftirqd/311:40:033
3949726993927,10cyclictest37-21ksoftirqd/310:07:053
394972699392,5cyclictest131rcu_preempt11:25:003
3949726993923,12cyclictest275-21dbus-daemon08:05:013
3949726993920,13cyclictest4000002-21modprobe10:45:003
3949726993920,13cyclictest4000002-21modprobe10:45:003
394972099394,3cyclictest131rcu_preempt11:37:381
394972099394,3cyclictest131rcu_preempt11:23:471
394972099393,4cyclictest150-21pr/ttyS212:28:241
394972099393,4cyclictest150-21pr/ttyS212:28:241
3949720993930,7cyclictest23-21ksoftirqd/110:24:371
3949720993927,10cyclictest23-21ksoftirqd/110:58:211
3949720993925,11cyclictest23-21ksoftirqd/109:15:541
394972099392,3cyclictest131rcu_preempt10:48:031
3949720993921,7cyclictest3956192-21ssh09:10:111
3949720993920,14cyclictest275-21dbus-daemon08:05:011
3949720993920,13cyclictest275-21dbus-daemon09:05:011
3949720993915,8cyclictest3998437-1kworker/u9:011:08:281
394972099391,3cyclictest131rcu_preempt11:35:011
394971699392,12cyclictest3954064-21munin-run08:35:000
3949726993829,7cyclictest37-21ksoftirqd/312:26:433
3949726993829,7cyclictest37-21ksoftirqd/312:26:433
394972699381,9cyclictest284-21systemd-logind09:45:013
3949726993814,18cyclictest3467602-21systemd-journal10:55:013
3949726993814,18cyclictest3467602-21systemd-journal10:55:013
3949726993811,5cyclictest131rcu_preempt07:13:583
3949726993810,5cyclictest131rcu_preempt07:57:303
3949723993830,5cyclictest3941960-21kworker/2:108:29:322
394972099387,6cyclictest131rcu_preempt09:10:021
394972099383,3cyclictest131rcu_preempt11:18:411
394972099383,3cyclictest131rcu_preempt11:18:411
3949720993828,7cyclictest3975683-21sshd09:52:241
3949720993821,12cyclictest275-21dbus-daemon12:35:001
394972099381,3cyclictest131rcu_preempt11:51:021
3949716993831,4cyclictest16950irq/224-800000007:50:000
3949716993810,7cyclictest131rcu_preempt07:10:020
394972699372,6cyclictest131rcu_preempt08:15:003
3949726993725,8cyclictest275-21dbus-daemon07:45:003
394972099374,4cyclictest131rcu_preempt11:40:191
394972099373,4cyclictest131rcu_preempt09:45:011
394972099372,4cyclictest131rcu_preempt11:26:271
394972099372,4cyclictest131rcu_preempt11:26:271
3949720993723,11cyclictest3951891-21cat07:50:021
3949720993722,12cyclictest23-21ksoftirqd/108:20:011
394972099371,7cyclictest4009359-21modprobe11:05:001
394971699371,5cyclictest3953802-21modprobe08:30:000
3949726993625,8cyclictest37-21ksoftirqd/307:50:013
3949726993624,10cyclictest37-21ksoftirqd/310:13:263
3949723993629,4cyclictest3941960-21kworker/2:107:22:462
3949723993628,4cyclictest3953823-21kworker/2:208:54:452
394972399361,4cyclictest131rcu_preempt07:30:012
3949720993624,8cyclictest3950166-21cron07:20:001
394972099362,3cyclictest131rcu_preempt07:40:001
3949720993623,9cyclictest1-21systemd07:55:011
3949720993623,9cyclictest1-21systemd07:55:001
3949716993626,7cyclictest3951971-21sshd07:54:450
3949716993626,7cyclictest3951971-21sshd07:54:440
3949726993527,5cyclictest3950441-21sh07:25:003
394972699352,5cyclictest131rcu_preempt08:45:023
3949726993523,10cyclictest37-21ksoftirqd/308:30:023
3949723993526,7cyclictest30-21ksoftirqd/208:30:022
394972099352,3cyclictest131rcu_preempt07:45:031
3949726993426,6cyclictest37-21ksoftirqd/307:10:023
394972099342,3cyclictest131rcu_preempt08:39:521
394972099331,4cyclictest131rcu_preempt08:31:241
394972099331,3cyclictest131rcu_preempt08:55:001
394972099325,3cyclictest131rcu_preempt07:30:011
394972699314,19cyclictest3954085-21grep08:35:013
394972099311,4cyclictest131rcu_preempt07:10:021
394972699301,4cyclictest131rcu_preempt09:05:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional