You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-21 - 00:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sat Jun 20, 2026 12:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
282052421470,5sleep02820529-21cpuspeed_turbos08:15:120
280328321210,2sleep20-21swapper/207:25:142
281142621150,8sleep234-21ksoftirqd/207:50:012
283377421080,4sleep13056-21PK-Backend08:50:221
279689027858,14sleep30-21swapper/307:09:443
139099780,67rtkit-daemon0-21swapper/007:05:550
28235632680,4sleep13056-21PK-Backend08:24:591
279670226344,13sleep20-21swapper/207:07:172
279669926140,14sleep10-21swapper/107:07:161
28619752570,4sleep30-21swapper/310:15:023
28172772550,1sleep20-21swapper/208:05:152
28715172460,4sleep13056-21PK-Backend10:40:011
28288102340,1chrt2828812-21dpkg08:40:002
28248442310,2sleep13056-21PK-Backend08:25:241
2797086993011,5cyclictest2804682-21cat07:30:113
279708699300,14cyclictest0-21swapper/311:40:123
279708699299,6cyclictest2814176-21awk07:55:203
2797076992910,17cyclictest2901323-21sshd12:04:160
139099290,4rtkit-daemon1389-21rtkit-daemon07:11:080
279708699289,6cyclictest2889700-21cat11:30:163
279708699288,6cyclictest2894458-21grep11:45:103
279708699284,14cyclictest2859579-21ps10:05:213
2797086992813,3cyclictest2839395-21sh09:10:003
279708699281,10cyclictest2832738-21apt08:50:123
2797086992810,5cyclictest2821999-21date08:20:013
279708699280,13cyclictest2869160-21unin-run10:35:003
279707699289,5cyclictest2803267-21cat07:25:140
279708699271,14cyclictest2848340-21apt09:35:113
2797086992710,5cyclictest2856734-21cron10:00:003
2797076992712,13cyclictest2897763-21apt-key11:55:010
2797076992711,14cyclictest2840475-21cut09:10:200
2797086992613,2cyclictest2843512-21diskstats09:20:133
2797086992612,3cyclictest2807503-21users07:35:233
279708699260,4cyclictest2892453-21perf11:40:003
279708699260,14cyclictest2900664-21grep12:00:183
279707699269,5cyclictest2895571-21sshd11:45:460
279707699267,5cyclictest2901978-21expr12:05:150
2797076992611,13cyclictest2849843-21grep09:40:010
2797076992610,4cyclictest2801035-21wc07:20:010
139099260,4rtkit-daemon1389-21rtkit-daemon12:25:071
139099260,4rtkit-daemon1389-21rtkit-daemon12:07:511
139099260,4rtkit-daemon1389-21rtkit-daemon08:14:501
279708699251,7cyclictest2815151-21apt08:00:133
2797086992511,3cyclictest2854572-21timerwakeupswit09:50:223
2797083992518,5cyclictest1894999-21snapd09:51:132
279707699259,3cyclictest2865659-21dump-pmu-power10:25:000
279707699257,5cyclictest2828728-21cron08:44:550
279707699257,4cyclictest2835925-21cron09:04:550
279707699256,5cyclictest2898966-21which11:55:200
279708699246,3cyclictest0-21swapper/308:25:143
279708699240,1cyclictest3056-21PK-Backend11:02:143
2797083992417,6cyclictest2860185-21cron10:10:002
2797081992421,2cyclictest3056-21PK-Backend08:35:011
279707699249,6cyclictest2863032-21load10:15:170
279707699249,5cyclictest2824672-21sensors08:25:220
279707699249,4cyclictest2835526-21sed08:55:230
279707699248,5cyclictest2907408-21irqstats12:20:160
279707699248,3cyclictest2909091-21if_enp2s012:25:140
279707699247,4cyclictest2802807-21wc07:25:010
279707699247,3cyclictest2812705-21sed07:50:210
279707699246,5cyclictest2870682-21sed10:35:230
279707699240,23cyclictest1389-21rtkit-daemon08:09:490
279707699240,1cyclictest0-21swapper/008:22:390
279708699231,7cyclictest2801228-21apt07:20:113
2797086992315,1cyclictest0-21swapper/310:27:403
279708699230,3cyclictest2845240-21df_inode09:25:143
279708699230,21cyclictest3056-21PK-Backend10:36:533
2797083992314,7cyclictest2882541-21expr11:10:122
279707699238,5cyclictest2859474-21sh10:05:210
279707699237,3cyclictest0-21swapper/011:30:440
279707699237,3cyclictest0-21swapper/011:24:190
279707699236,2cyclictest0-21swapper/008:30:480
279707699232,6cyclictest2156-21gnome-shell11:41:100
2797076992312,10cyclictest2828805-21apt-config08:40:010
279707699230,1cyclictest0-21swapper/007:55:420
279708699228,12cyclictest2804427-21latency_hist07:30:003
2797086992213,7cyclictest2872009-21df10:40:133
2797083992220,1cyclictest0-21swapper/211:03:332
2797083992220,1cyclictest0-21swapper/208:40:512
2797083992215,5cyclictest2904902-21cron12:14:592
2797083992215,5cyclictest2904902-21cron12:14:592
2797081992220,1cyclictest3056-21PK-Backend07:38:231
2797081992220,1cyclictest0-21swapper/109:30:391
2797081992219,2cyclictest3056-21PK-Backend10:18:031
2797081992219,2cyclictest3056-21PK-Backend07:54:121
2797081992215,5cyclictest2906910-21acpi12:20:111
2797081992213,7cyclictest2861936-21cron10:15:001
279708199220,21cyclictest0-21swapper/111:13:051
279707699226,5cyclictest2838650-21awk09:05:180
279707699226,2cyclictest28563892chrt09:55:450
279707699226,2cyclictest28112742chrt07:48:250
279707699224,5cyclictest883-21dbus-daemon11:20:000
279707699221,8cyclictest2883777-21awk11:15:000
279707699221,8cyclictest2860917-21/usr/sbin/munin10:10:160
279707699221,5cyclictest2818588-21/usr/sbin/munin08:10:120
279708699212,4cyclictest2808220-21apt07:40:133
279708699211,7cyclictest2827649-21df_inode08:35:143
279708699210,2cyclictest2825907-21df08:30:123
279708699210,19cyclictest3056-21PK-Backend12:17:453
279708699210,19cyclictest3056-21PK-Backend11:51:363
2797083992119,1cyclictest0-21swapper/212:15:322
2797083992119,1cyclictest0-21swapper/211:35:512
2797083992119,1cyclictest0-21swapper/211:22:472
2797083992119,1cyclictest0-21swapper/209:59:322
2797083992119,1cyclictest0-21swapper/209:03:562
2797083992119,1cyclictest0-21swapper/208:54:102
2797083992119,1cyclictest0-21swapper/208:28:542
2797083992119,1cyclictest0-21swapper/208:21:322
2797083992119,1cyclictest0-21swapper/208:16:512
2797083992119,1cyclictest0-21swapper/208:11:232
2797083992119,1cyclictest0-21swapper/208:03:192
2797083992119,1cyclictest0-21swapper/207:20:552
2797083992119,1cyclictest0-21swapper/207:18:582
2797083992119,1cyclictest0-21swapper/207:12:152
2797083992112,7cyclictest2873098-21cron10:45:002
279708399210,20cyclictest0-21swapper/211:30:172
279708399210,20cyclictest0-21swapper/210:57:252
279708399210,20cyclictest0-21swapper/209:17:482
279708399210,1cyclictest0-21swapper/211:25:172
2797081992119,1cyclictest3056-21PK-Backend12:38:101
2797081992119,1cyclictest3056-21PK-Backend11:34:361
2797081992119,1cyclictest3056-21PK-Backend10:35:061
2797081992119,1cyclictest3056-21PK-Backend09:36:191
2797081992119,1cyclictest3056-21PK-Backend07:49:591
2797081992119,1cyclictest3056-21PK-Backend07:35:001
2797081992119,1cyclictest0-21swapper/112:15:451
2797081992119,1cyclictest0-21swapper/112:01:521
2797081992119,1cyclictest0-21swapper/108:58:341
2797081992119,1cyclictest0-21swapper/108:18:521
2797081992119,1cyclictest0-21swapper/107:57:371
2797081992119,1cyclictest0-21swapper/107:14:101
2797081992118,2cyclictest3056-21PK-Backend07:44:111
2797081992118,2cyclictest3056-21PK-Backend07:17:141
279708199210,2cyclictest2882040-21latency_hist11:10:011
279708199210,20cyclictest3056-21PK-Backend10:32:031
279708199210,20cyclictest0-21swapper/112:11:181
279708199210,20cyclictest0-21swapper/112:11:181
279708199210,19cyclictest3056-21PK-Backend11:41:171
279708199210,19cyclictest3056-21PK-Backend10:28:241
279708199210,19cyclictest3056-21PK-Backend09:47:551
279708199210,19cyclictest3056-21PK-Backend08:06:151
279708199210,19cyclictest0-21swapper/110:56:361
279707699219,2cyclictest2880780-21cstates11:05:130
279707699218,2cyclictest2845303-21if_enp2s009:25:170
279707699211,8cyclictest2156-21gnome-shell11:27:090
2797076992118,2cyclictest3056-21PK-Backend10:45:010
279707699211,7cyclictest2842873-21/usr/sbin/munin09:20:000
2797076992116,4cyclictest13-21ksoftirqd/012:19:210
2797076992116,4cyclictest0-21swapper/009:54:590
279707699211,5cyclictest2904685-21lsb_release12:12:170
279707699211,5cyclictest2904685-21lsb_release12:12:170
279707699210,8cyclictest2853065-21chrt09:47:350
279707699210,19cyclictest3056-21PK-Backend09:40:530
139099210,3rtkit-daemon1389-21rtkit-daemon07:27:261
2797086992018,1cyclictest3056-21PK-Backend11:05:193
2797086992018,1cyclictest0-21swapper/308:42:103
2797086992018,1cyclictest0-21swapper/307:48:013
2797086992017,1cyclictest3056-21PK-Backend11:59:403
2797086992013,5cyclictest2847328-21ntp_kernel_pll_09:30:203
2797086992011,6cyclictest883-21dbus-daemon09:50:003
279708699200,7cyclictest2840615-21awk09:10:223
279708699200,1cyclictest0-21swapper/309:02:303
279708699200,19cyclictest3056-21PK-Backend08:14:593
279708699200,19cyclictest0-21swapper/311:23:443
279708699200,19cyclictest0-21swapper/311:18:103
279708699200,19cyclictest0-21swapper/308:20:053
279708699200,19cyclictest0-21swapper/308:06:063
279708699200,19cyclictest0-21swapper/307:14:563
2797083992018,1cyclictest3056-21PK-Backend12:02:482
2797083992018,1cyclictest0-21swapper/212:36:142
2797083992018,1cyclictest0-21swapper/212:31:002
2797083992018,1cyclictest0-21swapper/212:29:392
2797083992018,1cyclictest0-21swapper/211:55:522
2797083992018,1cyclictest0-21swapper/211:06:032
2797083992018,1cyclictest0-21swapper/210:51:522
2797083992018,1cyclictest0-21swapper/210:47:252
2797083992018,1cyclictest0-21swapper/210:24:342
2797083992018,1cyclictest0-21swapper/209:26:542
2797083992018,1cyclictest0-21swapper/209:08:562
2797083992018,1cyclictest0-21swapper/208:45:102
2797083992018,1cyclictest0-21swapper/207:36:492
2797083992018,1cyclictest0-21swapper/207:34:122
2797083992017,2cyclictest0-21swapper/212:24:252
2797083992015,4cyclictest978-21snmpd12:07:392
2797083992013,5cyclictest2866274-21expr10:25:142
279708399200,1cyclictest0-21swapper/211:47:342
279708399200,1cyclictest0-21swapper/210:36:162
279708399200,1cyclictest0-21swapper/210:12:032
279708399200,1cyclictest0-21swapper/208:57:472
279708399200,19cyclictest0-21swapper/210:16:292
279708399200,19cyclictest0-21swapper/209:46:082
279708399200,19cyclictest0-21swapper/207:52:522
2797081992018,1cyclictest3056-21PK-Backend12:32:211
2797081992018,1cyclictest3056-21PK-Backend11:37:391
2797081992018,1cyclictest3056-21PK-Backend11:27:521
2797081992018,1cyclictest3056-21PK-Backend09:55:121
2797081992018,1cyclictest3056-21PK-Backend09:51:331
2797081992018,1cyclictest3056-21PK-Backend09:05:161
2797081992018,1cyclictest3056-21PK-Backend08:46:581
2797081992018,1cyclictest0-21swapper/111:49:561
2797081992018,1cyclictest0-21swapper/111:03:191
2797081992017,2cyclictest3056-21PK-Backend09:28:221
2797081992014,4cyclictest1-21systemd11:54:191
279708199200,1cyclictest0-21swapper/110:20:331
279708199200,19cyclictest3056-21PK-Backend11:16:091
279708199200,19cyclictest3056-21PK-Backend09:19:291
279708199200,19cyclictest0-21swapper/110:05:321
279708199200,19cyclictest0-21swapper/107:22:491
279708199200,18cyclictest3056-21PK-Backend09:21:251
2797076992018,1cyclictest0-21swapper/012:30:060
2797076992018,1cyclictest0-21swapper/009:33:300
279707699200,1cyclictest0-21swapper/010:27:580
279707699200,19cyclictest0-21swapper/011:37:200
279707699200,19cyclictest0-21swapper/008:52:540
279707699200,19cyclictest0-21swapper/007:36:440
2797086991917,1cyclictest3056-21PK-Backend11:12:013
2797086991917,1cyclictest0-21swapper/312:31:513
2797086991917,1cyclictest0-21swapper/311:25:403
2797086991917,1cyclictest0-21swapper/310:20:583
2797086991917,1cyclictest0-21swapper/308:48:373
2797086991917,1cyclictest0-21swapper/307:18:003
2797086991912,5cyclictest436-20systemd-journal10:04:553
2797086991911,6cyclictest436-20systemd-journal12:39:553
279708699190,1cyclictest0-21swapper/310:56:063
279708699190,1cyclictest0-21swapper/307:52:133
279708699190,18cyclictest3056-21PK-Backend12:22:453
279708699190,18cyclictest3056-21PK-Backend12:12:453
279708699190,18cyclictest3056-21PK-Backend12:12:453
279708699190,18cyclictest0-21swapper/310:45:313
279708699190,18cyclictest0-21swapper/310:19:023
279708699190,18cyclictest0-21swapper/309:17:303
2797083991917,1cyclictest0-21swapper/209:10:522
2797083991917,1cyclictest0-21swapper/207:44:262
2797083991913,5cyclictest2885090-21cat11:15:212
279708399190,1cyclictest0-21swapper/211:52:002
279708399190,1cyclictest0-21swapper/210:33:462
279708399190,1cyclictest0-21swapper/207:55:282
279708399190,18cyclictest0-21swapper/211:42:002
2797081991917,1cyclictest3056-21PK-Backend11:22:181
2797081991917,1cyclictest3056-21PK-Backend10:50:271
2797081991917,1cyclictest3056-21PK-Backend10:47:571
2797081991917,1cyclictest3056-21PK-Backend09:11:251
2797081991917,1cyclictest3056-21PK-Backend08:30:571
279708199190,1cyclictest3056-21PK-Backend08:41:161
279708199190,1cyclictest3056-21PK-Backend08:01:021
279708199190,1cyclictest0-21swapper/111:57:261
279707699197,3cyclictest28314102sleep008:45:160
279707699192,2cyclictest141rcu_preempt10:55:010
279707699191,8cyclictest2844000-21python309:20:190
279707699191,7cyclictest2808696-21irqstats07:40:180
2797076991917,1cyclictest0-21swapper/010:41:230
279707699190,7cyclictest2879708-21/usr/sbin/munin11:00:210
279707699190,7cyclictest2868035-21diskstats10:30:130
279707699190,18cyclictest0-21swapper/012:35:060
279707699190,18cyclictest0-21swapper/010:02:160
279707699190,18cyclictest0-21swapper/008:03:190
279708699188,7cyclictest883-21dbus-daemon12:29:233
2797086991811,6cyclictest2850483-21ls09:40:183
2797086991811,5cyclictest2901770-21cat12:05:143
279708699180,16cyclictest2834838-21df_inode08:55:143
2797083991814,3cyclictest2846749-21cpuspeed_turbos09:30:142
2797086991712,3cyclictest2876055-21threads10:50:223
2797083991710,6cyclictest2826995-21cron08:34:592
2797083991710,5cyclictest2848556-21cat09:35:122
2797081991715,1cyclictest3056-21PK-Backend10:02:281
2797081991711,4cyclictest2850437-21df09:40:151
2797081991710,5cyclictest2836735-21sed09:00:171
279708399169,5cyclictest2850264-21grep09:40:142
2797083991614,1cyclictest0-21swapper/209:21:072
2797083991610,5cyclictest2857964-21sed10:00:222
2797076991612,3cyclictest2878577-21munin-plugin-st11:00:000
279707699140,1cyclictest3056-21PK-Backend07:30:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional