You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 07:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 17, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
246506821120,6sleep1251rcuc/120:05:131
1973991040,5rtkit-daemon0-21swapper/221:17:173
255291921030,5sleep20-21swapper/200:15:132
1973991020,6rtkit-daemon4497-21PK-Backend21:11:413
1973991020,5rtkit-daemon0-21swapper/220:17:333
1973991010,5rtkit-daemon4497-21PK-Backend00:03:423
1973991010,5rtkit-daemon0-21swapper/221:28:563
197399950,6rtkit-daemon4497-21PK-Backend23:00:143
244523499862,6cyclictest0-21swapper/021:23:580
25522812800,7sleep1974-21in:imuxsock00:14:591
25493502760,4sleep10-21swapper/100:05:131
244364527645,22sleep20-21swapper/219:05:112
244523499741,50cyclictest2542006-21cat23:45:020
197399740,6rtkit-daemon4497-21PK-Backend21:44:532
197399740,6rtkit-daemon4497-21PK-Backend00:08:282
197399740,5rtkit-daemon0-21swapper/200:33:342
197399730,5rtkit-daemon4497-21PK-Backend00:04:412
197399730,5rtkit-daemon0-21swapper/221:30:562
197399730,5rtkit-daemon0-21swapper/219:58:442
197399720,6rtkit-daemon4497-21PK-Backend21:48:312
197399720,6rtkit-daemon0-21swapper/220:37:292
197399720,5rtkit-daemon0-21swapper/220:06:282
197399720,5rtkit-daemon0-21swapper/219:44:522
197399720,5rtkit-daemon0-21swapper/200:37:332
244434827142,20sleep00-21swapper/019:05:210
197399710,6rtkit-daemon4497-21PK-Backend22:24:232
197399710,5rtkit-daemon4497-21PK-Backend23:58:012
197399710,5rtkit-daemon4497-21PK-Backend23:38:152
197399710,5rtkit-daemon4497-21PK-Backend19:48:342
197399710,5rtkit-daemon0-21swapper/221:52:372
197399710,5rtkit-daemon0-21swapper/221:18:092
197399710,5rtkit-daemon0-21swapper/220:45:162
197399700,5rtkit-daemon0-21swapper/221:56:492
197399700,5rtkit-daemon0-21swapper/221:37:092
197399700,5rtkit-daemon0-21swapper/219:28:482
197399690,5rtkit-daemon0-21swapper/221:26:172
197399690,5rtkit-daemon0-21swapper/220:03:552
197399680,4rtkit-daemon0-21swapper/220:58:222
2445244996721,9cyclictest34-21ksoftirqd/223:28:002
244523499672,48cyclictest2546199-21memory23:55:220
244523499661,39cyclictest2487797-21sed21:10:020
244502426541,18sleep30-21swapper/319:09:103
244523499641,37cyclictest2536328-21taskset23:26:080
244523499641,37cyclictest2523546-21/usr/sbin/munin22:50:260
244523499641,37cyclictest2523546-21/usr/sbin/munin22:50:260
197399640,10rtkit-daemon1972-21rtkit-daemon22:39:431
244523499631,41cyclictest2534580-21taskset23:20:550
244523499622,14cyclictest2069-21gnome-shell23:06:580
244523499621,38cyclictest2505160-21taskset21:59:510
244496926243,14sleep10-21swapper/119:08:271
244523499611,37cyclictest2470436-21cstates20:20:140
25081862600,5sleep24497-21PK-Backend22:05:302
244524899602,16cyclictest2069-21gnome-shell00:32:593
244523699602,16cyclictest2069-21gnome-shell19:46:591
244523699602,15cyclictest2069-21gnome-shell22:17:591
244523699592,53cyclictest2069-21gnome-shell21:40:581
244523499592,15cyclictest2069-21gnome-shell22:30:590
244523499591,40cyclictest924-21NetworkManager22:38:530
24837172580,6chrt4497-21PK-Backend20:58:581
2445244995814,8cyclictest34-21ksoftirqd/222:11:272
244523499587,29cyclictest2511702-21timerwakeupswit22:15:280
244523499582,52cyclictest2069-21gnome-shell23:52:580
197399580,8rtkit-daemon1972-21rtkit-daemon22:46:170
244524899572,51cyclictest2069-21gnome-shell20:41:593
244524899572,50cyclictest2069-21gnome-shell00:27:583
2445236995714,29cyclictest26-21ksoftirqd/100:00:591
244523499572,12cyclictest2510426-21kworker/0:223:37:590
244523499571,51cyclictest2491158-21latency_hist21:20:010
244523499571,50cyclictest2480765-21grep20:50:140
244523499571,50cyclictest2480765-21grep20:50:140
197399570,8rtkit-daemon1972-21rtkit-daemon00:23:260
25143972560,6sleep2331rcuc/222:25:142
25143972560,6sleep2331rcuc/222:25:142
244523499561,50cyclictest2458600-21latency19:45:210
2445244995548,5cyclictest0-21swapper/221:23:592
2445244995514,7cyclictest34-21ksoftirqd/222:30:272
2445244995513,8cyclictest34-21ksoftirqd/223:15:262
2445244995513,8cyclictest34-21ksoftirqd/223:15:252
2445244995512,8cyclictest34-21ksoftirqd/221:12:482
244523699552,48cyclictest2069-21gnome-shell23:13:591
244523699552,10cyclictest2069-21gnome-shell21:48:591
2445234995520,30cyclictest2532668-21find23:15:300
2445234995520,30cyclictest2532668-21find23:15:300
244523499551,30cyclictest438-20systemd-journal21:36:560
2445248995425,15cyclictest42-21ksoftirqd/322:11:273
244523499542,48cyclictest2069-21gnome-shell19:30:590
244524899532,10cyclictest2069-21gnome-shell20:32:593
244524499532,10cyclictest2069-21gnome-shell20:14:592
244523699532,47cyclictest2069-21gnome-shell22:03:591
244523499532,47cyclictest2069-21gnome-shell19:11:590
244523499532,29cyclictest2460176-21df19:50:150
244523499531,34cyclictest0-21swapper/022:00:190
244524899528,31cyclictest141rcu_preempt19:30:233
244524899524,8cyclictest53-21kcompactd020:53:593
244524899524,8cyclictest53-21kcompactd020:53:593
244524899522,46cyclictest2069-21gnome-shell21:01:593
244524499522,46cyclictest2069-21gnome-shell20:26:592
244524499522,46cyclictest2069-21gnome-shell20:26:592
244523699522,46cyclictest2069-21gnome-shell23:48:581
244523699522,46cyclictest2069-21gnome-shell22:05:591
244523699522,45cyclictest2069-21gnome-shell19:33:581
244523499521,46cyclictest2464900-21latency_hist20:05:020
244523499521,45cyclictest2509470-21latency22:10:200
244523499521,33cyclictest0-21swapper/023:45:270
25451712510,4sleep30-21swapper/323:52:563
244524899512,45cyclictest2069-21gnome-shell20:24:593
2445236995139,8cyclictest2069-21gnome-shell22:40:591
244523699512,45cyclictest2069-21gnome-shell22:55:581
2445234995140,7cyclictest2069-21gnome-shell22:07:590
244523499512,45cyclictest2069-21gnome-shell00:04:590
244523499511,46cyclictest2069-21gnome-shell23:30:580
244523499511,45cyclictest2489399-21cat21:15:020
244523499511,30cyclictest0-21swapper/020:37:080
25278132500,14sleep126-21ksoftirqd/123:02:571
2445248995035,11cyclictest2069-21gnome-shell20:45:593
2445248995022,12cyclictest42-21ksoftirqd/322:30:273
2445244995039,6cyclictest25279732chrt23:04:592
2445244995017,28cyclictest34-21ksoftirqd/219:21:582
244523499502,43cyclictest2551171-21expr00:10:150
244523499501,30cyclictest0-21swapper/000:28:420
24734952490,27sleep34497-21PK-Backend20:30:023
24734952490,27sleep34497-21PK-Backend20:30:023
244524899492,5cyclictest4497-21PK-Backend19:42:583
244524499498,36cyclictest2069-21gnome-shell21:04:592
244524499492,7cyclictest4497-21PK-Backend19:19:592
2445244994922,8cyclictest34-21ksoftirqd/222:55:142
244524499491,45cyclictest2069-21gnome-shell21:06:582
2445236994937,9cyclictest26-21ksoftirqd/119:14:131
244523699492,44cyclictest0-21swapper/122:49:591
244523499494,21cyclictest13-21ksoftirqd/020:47:010
2445234994938,7cyclictest2069-21gnome-shell00:06:590
2445234994934,11cyclictest2069-21gnome-shell21:44:590
244523499492,7cyclictest0-21swapper/019:40:580
2445234994923,22cyclictest2557891-21cpuspeed_turbos00:30:150
244523499491,44cyclictest0-21swapper/020:00:020
244523499491,43cyclictest2451292-21cut19:25:140
244523499491,28cyclictest0-21swapper/020:05:420
244524899482,6cyclictest0-21swapper/323:07:583
2445248994822,9cyclictest42-21ksoftirqd/320:14:593
244524499482,6cyclictest0-21swapper/222:42:592
244523699482,5cyclictest0-21swapper/100:22:591
244523699482,43cyclictest4497-21PK-Backend23:16:591
244523699482,43cyclictest4497-21PK-Backend23:16:581
244523699481,6cyclictest4497-21PK-Backend20:12:591
244523699481,6cyclictest4497-21PK-Backend00:38:581
2445234994840,5cyclictest0-21swapper/019:23:590
244523499482,43cyclictest0-21swapper/021:04:590
244523499482,43cyclictest0-21swapper/020:40:590
2445234994822,22cyclictest2559668-21tr00:35:140
244523499481,42cyclictest2552653-21grep00:15:110
244523499481,41cyclictest2525068-21cat22:55:150
244524899478,26cyclictest0-21swapper/323:47:443
244524899472,6cyclictest0-21swapper/320:36:583
244524899472,41cyclictest4497-21PK-Backend22:49:583
244524899471,41cyclictest4497-21PK-Backend00:22:583
2445244994739,5cyclictest0-21swapper/219:11:592
244524499472,5cyclictest0-21swapper/223:45:592
244524499472,42cyclictest0-21swapper/220:20:582
244524499471,5cyclictest4497-21PK-Backend22:46:592
2445236994740,4cyclictest0-21swapper/121:03:591
2445236994736,7cyclictest2069-21gnome-shell21:53:591
244523699472,5cyclictest0-21swapper/122:34:581
244523699472,43cyclictest0-21swapper/119:29:591
2445236994713,5cyclictest26-21ksoftirqd/121:27:591
2445234994739,5cyclictest0-21swapper/020:28:590
2445234994739,5cyclictest0-21swapper/020:28:590
244523499472,40cyclictest2530738-21ntp_states23:10:250
2445234994716,26cyclictest923-21dbus-daemon20:14:140
244523499471,41cyclictest2513048-21needreboot22:20:220
244523499471,41cyclictest2502297-21expr21:50:140
244523499471,41cyclictest2501779-21cat21:50:000
2445234994711,32cyclictest0-21swapper/021:25:130
24548732460,13sleep2341ksoftirqd/219:35:162
244524899466,35cyclictest2069-21gnome-shell19:16:583
244524899462,5cyclictest0-21swapper/321:48:593
244524899462,5cyclictest0-21swapper/321:08:593
244524899462,5cyclictest0-21swapper/300:08:593
244524899462,42cyclictest0-21swapper/322:05:593
244524899461,42cyclictest4497-21PK-Backend23:13:583
244524899461,42cyclictest4497-21PK-Backend19:50:593
244524899461,42cyclictest4497-21PK-Backend19:25:593
2445244994640,4cyclictest0-21swapper/220:40:592
2445244994636,7cyclictest2069-21gnome-shell23:20:592
244524499461,42cyclictest4497-21PK-Backend00:20:592
244524499461,41cyclictest4497-21PK-Backend23:05:592
244524499461,12cyclictest0-21swapper/220:20:022
2445236994639,4cyclictest4497-21PK-Backend20:36:591
2445236994639,4cyclictest0-21swapper/121:16:581
2445236994638,5cyclictest0-21swapper/120:26:581
2445236994638,5cyclictest0-21swapper/120:26:581
2445236994637,6cyclictest231irq_work/119:19:591
244523699461,41cyclictest4497-21PK-Backend20:46:591
244523499466,37cyclictest1137-21runrttasks19:37:020
2445234994640,4cyclictest0-21swapper/022:28:590
2445234994640,4cyclictest0-21swapper/022:28:590
244523499463,38cyclictest2468687-21tr20:15:150
2445234994612,29cyclictest2447389-21ntpq19:15:050
244524899452,41cyclictest0-21swapper/322:03:593
244524499457,14cyclictest34-21ksoftirqd/222:15:142
2445244994537,5cyclictest0-21swapper/223:12:582
244524499452,41cyclictest0-21swapper/223:43:582
2445244994522,18cyclictest2537709-21/usr/sbin/munin23:30:252
244524499451,5cyclictest4497-21PK-Backend00:12:592
2445236994539,4cyclictest0-21swapper/120:31:591
2445234994517,24cyclictest2526738-21cstates23:00:140
2445234994512,28cyclictest2482611-21expr20:55:140
197399450,5rtkit-daemon4497-21PK-Backend23:33:093
244524899448,23cyclictest42-21ksoftirqd/319:10:113
2445248994437,4cyclictest4497-21PK-Backend23:25:583
2445248994436,5cyclictest4497-21PK-Backend22:41:583
244524499443,21cyclictest34-21ksoftirqd/219:55:012
244523499441,38cyclictest2496754-21latency_hist21:35:020
2445248994337,4cyclictest0-21swapper/321:31:593
2445248994336,4cyclictest4497-21PK-Backend22:52:593
2445248994336,4cyclictest4497-21PK-Backend22:52:593
2445248994322,9cyclictest42-21ksoftirqd/321:55:003
2445244994337,4cyclictest0-21swapper/222:36:592
2445244994335,5cyclictest0-21swapper/219:31:592
244523699432,36cyclictest2069-21gnome-shell19:54:591
2445236994323,15cyclictest26-21ksoftirqd/100:25:291
2445236994323,11cyclictest26-21ksoftirqd/121:05:101
2445236994316,5cyclictest141rcu_preempt23:28:021
244523499431,37cyclictest2474008-21cstates20:30:130
244524899429,18cyclictest141rcu_preempt20:55:283
244524899428,19cyclictest141rcu_preempt23:15:243
244524899428,19cyclictest141rcu_preempt23:15:243
2445248994221,8cyclictest42-21ksoftirqd/319:35:233
244524899421,7cyclictest2069-21gnome-shell19:21:593
2445236994231,7cyclictest2069-21gnome-shell19:39:591
2445236994225,13cyclictest26-21ksoftirqd/120:42:361
2445236994224,14cyclictest26-21ksoftirqd/123:54:111
2445236994224,14cyclictest26-21ksoftirqd/123:23:231
2445236994224,14cyclictest26-21ksoftirqd/122:20:291
2445236994224,14cyclictest26-21ksoftirqd/119:56:221
2445236994224,14cyclictest26-21ksoftirqd/119:21:311
2445236994223,14cyclictest26-21ksoftirqd/122:25:281
2445236994223,14cyclictest26-21ksoftirqd/122:25:281
2445236994223,14cyclictest26-21ksoftirqd/121:20:191
244523499421,37cyclictest0-21swapper/022:40:130
24623432410,6sleep3411rcuc/319:55:253
2445248994123,6cyclictest42-21ksoftirqd/321:55:373
2445248994118,18cyclictest2069-21gnome-shell00:18:593
2445248994118,10cyclictest42-21ksoftirqd/300:10:163
2445248994113,9cyclictest42-21ksoftirqd/323:00:003
2445236994124,13cyclictest26-21ksoftirqd/123:34:071
2445236994123,6cyclictest26-21ksoftirqd/122:51:481
2445236994123,6cyclictest26-21ksoftirqd/122:51:481
2445236994123,14cyclictest26-21ksoftirqd/123:55:311
2445236994123,14cyclictest26-21ksoftirqd/123:08:511
2445236994123,14cyclictest26-21ksoftirqd/122:13:251
2445236994123,14cyclictest26-21ksoftirqd/100:32:141
2445236994123,14cyclictest26-21ksoftirqd/100:15:111
2445236994122,14cyclictest26-21ksoftirqd/120:00:101
25335752400,6chrt4497-21PK-Backend23:20:133
2445248994034,4cyclictest0-21swapper/322:16:593
2445248994023,6cyclictest42-21ksoftirqd/323:39:493
2445248994018,9cyclictest42-21ksoftirqd/321:40:153
2445248994012,16cyclictest42-21ksoftirqd/321:35:123
2445236994023,6cyclictest26-21ksoftirqd/119:40:491
2445236994023,5cyclictest26-21ksoftirqd/123:40:201
2445236994023,5cyclictest26-21ksoftirqd/121:59:171
2445236994023,13cyclictest26-21ksoftirqd/123:36:401
2445236994022,6cyclictest26-21ksoftirqd/121:35:141
244524899398,13cyclictest141rcu_preempt00:40:013
2445248993932,4cyclictest4497-21PK-Backend20:04:593
2445248993920,11cyclictest42-21ksoftirqd/322:30:003
2445248993920,11cyclictest42-21ksoftirqd/322:30:003
2445244993931,5cyclictest4497-21PK-Backend00:29:582
244524499391,11cyclictest4497-21PK-Backend22:00:282
2445236993922,5cyclictest26-21ksoftirqd/121:30:091
2445236993922,5cyclictest26-21ksoftirqd/121:10:261
2445236993922,5cyclictest26-21ksoftirqd/120:22:241
244523699391,32cyclictest2401452-21kworker/1:120:18:591
2445248993813,15cyclictest141rcu_preempt23:40:233
244524499385,16cyclictest2523562-21ps22:50:252
244524499385,16cyclictest2523562-21ps22:50:252
244524499383,19cyclictest34-21ksoftirqd/220:30:212
244524499381,10cyclictest4497-21PK-Backend20:55:002
244524499381,10cyclictest4497-21PK-Backend20:55:002
2445236993821,5cyclictest26-21ksoftirqd/120:50:191
2445236993821,5cyclictest26-21ksoftirqd/120:50:181
2445248993722,12cyclictest42-21ksoftirqd/322:35:143
2445248993719,14cyclictest42-21ksoftirqd/300:00:013
244524499371,12cyclictest438-20systemd-journal23:55:012
244524899366,9cyclictest42-21ksoftirqd/320:05:303
2445248993619,13cyclictest42-21ksoftirqd/322:20:123
2445248993518,13cyclictest42-21ksoftirqd/319:47:243
2445248993516,7cyclictest42-21ksoftirqd/321:24:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional