You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 20:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Mon Mar 23, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
283904221290,4sleep02839044-21tr10:45:110
276279127838,14sleep20-21swapper/207:09:092
139099780,63rtkit-daemon0-21swapper/307:07:503
276284227540,13sleep10-21swapper/107:09:481
276254026445,13sleep00-21swapper/007:05:520
28282702500,3sleep20-21swapper/210:15:012
27800722290,4sleep33056-21PK-Backend07:55:153
276304599270,25cyclictest2824834-21latency_hist10:05:010
139099270,4rtkit-daemon1389-21rtkit-daemon10:30:351
28444392250,9sleep342-21ksoftirqd/311:00:133
27939712250,2sleep23056-21PK-Backend08:35:172
27939712250,2sleep23056-21PK-Backend08:35:172
276305499241,14cyclictest141rcu_preempt07:21:263
2763054992410,3cyclictest141rcu_preempt12:35:283
2763051992410,12cyclictest2777638-21unin-run07:50:002
2763045992413,10cyclictest2844041-21/usr/sbin/munin11:00:120
276304599240,23cyclictest1389-21rtkit-daemon09:21:470
27980372230,3sleep1251rcuc/108:45:221
27867262230,7sleep20-21swapper/208:15:122
27639062230,3sleep10-21swapper/107:10:151
2763054992315,6cyclictest2782764-21sshd08:04:263
276305499230,21cyclictest3056-21PK-Backend09:26:413
2763051992321,1cyclictest0-21swapper/211:36:402
276305199230,3cyclictest2637-21gmain11:51:412
276305199230,21cyclictest2804651-21grep09:05:192
2763046992314,7cyclictest2833249-21ntp10:28:281
276304699230,21cyclictest2875665-21date12:30:011
2763045992321,1cyclictest0-21swapper/012:38:250
2763045992313,7cyclictest436-20systemd-journal07:54:590
2763054992220,1cyclictest3056-21PK-Backend07:28:043
2763054992219,2cyclictest3056-21PK-Backend09:06:463
2763054992217,4cyclictest42-21ksoftirqd/308:30:003
2763054992211,10cyclictest2837870-21ntp_states10:40:183
276305499220,20cyclictest3056-21PK-Backend08:57:553
276305499220,20cyclictest3056-21PK-Backend08:43:073
2763051992220,1cyclictest0-21swapper/210:24:202
2763046992215,5cyclictest883-21dbus-daemon11:10:011
276304699220,3cyclictest2802926-21ntp_states09:00:191
276304699220,2cyclictest2820670-21sed09:50:211
276304699220,21cyclictest0-21swapper/109:40:211
276304599223,15cyclictest2801922-21perf09:00:000
2763045992220,1cyclictest0-21swapper/011:33:230
2763045992220,1cyclictest0-21swapper/011:33:220
2763045992220,1cyclictest0-21swapper/011:13:420
2763045992220,1cyclictest0-21swapper/010:21:240
2763045992220,1cyclictest0-21swapper/009:56:090
2763045992220,1cyclictest0-21swapper/009:48:040
2763045992220,1cyclictest0-21swapper/009:16:140
2763045992220,1cyclictest0-21swapper/007:26:090
139099220,4rtkit-daemon1389-21rtkit-daemon12:18:450
28637142210,4sleep00-21swapper/011:55:120
2763054992119,1cyclictest3056-21PK-Backend12:18:103
2763054992119,1cyclictest3056-21PK-Backend11:05:023
2763054992119,1cyclictest3056-21PK-Backend09:46:013
2763054992119,1cyclictest0-21swapper/310:52:053
2763054992119,1cyclictest0-21swapper/308:23:063
2763054992119,1cyclictest0-21swapper/308:12:193
2763054992118,2cyclictest3056-21PK-Backend10:59:423
2763054992118,2cyclictest3056-21PK-Backend10:49:483
2763054992118,2cyclictest3056-21PK-Backend10:23:463
2763054992118,2cyclictest3056-21PK-Backend08:54:093
2763054992118,2cyclictest3056-21PK-Backend08:39:363
2763054992118,2cyclictest3056-21PK-Backend08:39:363
2763054992118,2cyclictest3056-21PK-Backend08:32:473
2763054992116,4cyclictest42-21ksoftirqd/307:40:003
276305499210,20cyclictest3056-21PK-Backend07:33:253
276305499210,20cyclictest3056-21PK-Backend07:33:243
276305499210,20cyclictest0-21swapper/312:33:503
276305499210,20cyclictest0-21swapper/310:02:443
276305499210,20cyclictest0-21swapper/309:13:013
276305499210,19cyclictest3056-21PK-Backend12:25:053
276305499210,19cyclictest3056-21PK-Backend10:05:343
276305499210,19cyclictest3056-21PK-Backend09:30:403
2763051992119,1cyclictest0-21swapper/212:26:152
2763051992119,1cyclictest0-21swapper/212:24:132
2763051992119,1cyclictest0-21swapper/212:13:102
2763051992119,1cyclictest0-21swapper/210:55:242
2763051992119,1cyclictest0-21swapper/210:40:432
2763051992119,1cyclictest0-21swapper/209:21:132
2763051992119,1cyclictest0-21swapper/209:10:132
2763051992119,1cyclictest0-21swapper/209:03:572
2763051992119,1cyclictest0-21swapper/208:45:452
2763051992119,1cyclictest0-21swapper/208:42:482
2763051992119,1cyclictest0-21swapper/208:20:172
2763051992119,1cyclictest0-21swapper/208:12:202
2763051992119,1cyclictest0-21swapper/207:41:102
2763051992119,1cyclictest0-21swapper/207:15:552
2763051992111,8cyclictest883-21dbus-daemon08:03:252
276305199210,20cyclictest0-21swapper/211:34:242
276305199210,20cyclictest0-21swapper/211:34:232
276305199210,20cyclictest0-21swapper/209:54:122
276305199210,20cyclictest0-21swapper/207:29:482
2763046992114,5cyclictest2869925-21sshd12:13:241
2763046992114,5cyclictest2785215-21fw_packets08:10:131
276304599219,8cyclictest883-21dbus-daemon09:40:000
276304599219,8cyclictest436-20systemd-journal08:55:000
2763045992119,1cyclictest0-21swapper/012:06:490
2763045992119,1cyclictest0-21swapper/012:03:440
2763045992119,1cyclictest0-21swapper/011:46:190
2763045992119,1cyclictest0-21swapper/010:28:200
2763045992119,1cyclictest0-21swapper/010:18:190
2763045992119,1cyclictest0-21swapper/009:30:540
2763045992119,1cyclictest0-21swapper/009:13:170
2763045992119,1cyclictest0-21swapper/008:32:480
2763045992119,1cyclictest0-21swapper/008:09:430
2763045992119,1cyclictest0-21swapper/008:02:190
276304599211,6cyclictest2812570-21perf09:30:000
2763045992114,5cyclictest2796701-21/usr/sbin/munin08:45:000
2763045992114,5cyclictest2156-21gnome-shell11:36:090
2763045992113,6cyclictest436-20systemd-journal07:35:000
2763045992113,6cyclictest436-20systemd-journal07:34:590
2763045992112,7cyclictest2803508-21sshd09:03:260
2763045992111,7cyclictest436-20systemd-journal11:20:010
276304599210,20cyclictest0-21swapper/007:36:030
276304599210,20cyclictest0-21swapper/007:13:180
276304599210,1cyclictest0-21swapper/011:53:290
276304599210,1cyclictest0-21swapper/008:21:580
276304599210,19cyclictest3056-21PK-Backend08:28:080
139099210,15rtkit-daemon3056-21PK-Backend08:17:173
139099210,13rtkit-daemon3056-21PK-Backend07:52:303
27828342200,3sleep12782827-21turbostat08:05:001
276305499206,13cyclictest2873784-21cron12:24:593
2763054992018,1cyclictest3056-21PK-Backend12:05:193
2763054992018,1cyclictest3056-21PK-Backend11:59:093
2763054992018,1cyclictest3056-21PK-Backend10:25:493
2763054992018,1cyclictest3056-21PK-Backend10:13:453
2763054992018,1cyclictest3056-21PK-Backend09:24:183
2763054992018,1cyclictest0-21swapper/307:47:243
2763054992018,1cyclictest0-21swapper/307:44:473
2763054992017,2cyclictest3056-21PK-Backend12:02:153
2763054992017,2cyclictest3056-21PK-Backend10:16:493
2763054992017,2cyclictest3056-21PK-Backend09:44:113
2763054992013,5cyclictest2834746-21sshd10:30:273
276305499200,1cyclictest3056-21PK-Backend12:11:143
276305499200,1cyclictest3056-21PK-Backend11:11:113
276305499200,1cyclictest3056-21PK-Backend07:14:193
276305499200,19cyclictest3056-21PK-Backend10:37:373
276305499200,19cyclictest3056-21PK-Backend10:37:373
276305499200,19cyclictest3056-21PK-Backend09:53:043
276305499200,19cyclictest0-21swapper/308:05:363
2763051992018,1cyclictest0-21swapper/212:16:152
2763051992018,1cyclictest0-21swapper/211:25:592
2763051992018,1cyclictest0-21swapper/210:31:312
2763051992018,1cyclictest0-21swapper/209:47:092
2763051992018,1cyclictest0-21swapper/209:31:492
2763051992018,1cyclictest0-21swapper/208:30:172
2763051992018,1cyclictest0-21swapper/208:27:002
276305199200,2cyclictest0-21swapper/210:50:512
276305199200,1cyclictest0-21swapper/207:11:572
276305199200,19cyclictest0-21swapper/212:09:042
276305199200,19cyclictest0-21swapper/211:12:262
276305199200,19cyclictest0-21swapper/210:37:262
276305199200,19cyclictest0-21swapper/210:37:262
276305199200,19cyclictest0-21swapper/210:27:252
276305199200,19cyclictest0-21swapper/210:19:142
276305199200,19cyclictest0-21swapper/210:03:532
276305199200,19cyclictest0-21swapper/209:56:022
276305199200,19cyclictest0-21swapper/209:35:272
276305199200,19cyclictest0-21swapper/208:57:012
276305199200,19cyclictest0-21swapper/207:36:092
2763046992018,1cyclictest0-21swapper/110:49:431
2763046992017,2cyclictest0-21swapper/111:15:181
2763046992017,2cyclictest0-21swapper/109:05:131
2763046992013,5cyclictest2871272-21netstat12:15:181
2763046992012,6cyclictest2877535-21awk12:35:121
276304699200,1cyclictest3056-21PK-Backend07:42:461
276304699200,19cyclictest3056-21PK-Backend09:12:091
276304699200,19cyclictest3056-21PK-Backend07:46:301
276304699200,19cyclictest0-21swapper/111:57:161
276304699200,19cyclictest0-21swapper/107:18:531
2763045992018,1cyclictest3056-21PK-Backend07:48:270
2763045992018,1cyclictest0-21swapper/011:42:010
2763045992018,1cyclictest0-21swapper/010:35:160
2763045992018,1cyclictest0-21swapper/010:35:150
2763045992012,6cyclictest2805376-21cron09:10:000
276304599200,19cyclictest0-21swapper/012:22:430
276304599200,19cyclictest0-21swapper/010:58:550
276304599200,19cyclictest0-21swapper/007:56:440
139099200,14rtkit-daemon3056-21PK-Backend09:57:083
2763054991917,1cyclictest3056-21PK-Backend09:01:063
276305199196,3cyclictest2845543-21cut11:05:012
2763051991917,1cyclictest0-21swapper/212:04:592
2763051991917,1cyclictest0-21swapper/211:47:362
2763051991917,1cyclictest0-21swapper/209:42:442
2763051991917,1cyclictest0-21swapper/209:25:132
2763051991917,1cyclictest0-21swapper/208:53:152
2763051991917,1cyclictest0-21swapper/207:51:382
2763051991917,1cyclictest0-21swapper/207:32:592
2763051991917,1cyclictest0-21swapper/207:32:592
2763051991912,5cyclictest2783537-21cat08:05:162
276305199190,1cyclictest3056-21PK-Backend09:19:382
276305199190,1cyclictest0-21swapper/211:41:002
276305199190,1cyclictest0-21swapper/210:48:342
276305199190,1cyclictest0-21swapper/207:20:222
276305199190,18cyclictest0-21swapper/212:35:132
276305199190,18cyclictest0-21swapper/207:56:112
2763046991917,1cyclictest0-21swapper/112:33:391
2763046991917,1cyclictest0-21swapper/111:51:481
2763046991917,1cyclictest0-21swapper/110:56:251
2763046991917,1cyclictest0-21swapper/110:36:181
2763046991917,1cyclictest0-21swapper/110:36:181
2763046991917,1cyclictest0-21swapper/109:15:271
2763046991917,1cyclictest0-21swapper/108:18:491
2763046991917,1cyclictest0-21swapper/107:24:541
2763046991912,5cyclictest2814945-21df_inode09:35:131
2763046991912,5cyclictest2811812-21ntp_kernel_err09:25:171
2763046991911,6cyclictest2809768-21fw_packets09:20:141
276304699190,1cyclictest3056-21PK-Backend11:41:131
276304699190,1cyclictest3056-21PK-Backend08:27:411
276304699190,1cyclictest0-21swapper/112:09:121
276304699190,1cyclictest0-21swapper/111:04:441
276304699190,18cyclictest0-21swapper/112:01:561
276304699190,18cyclictest0-21swapper/109:31:551
276304699190,18cyclictest0-21swapper/108:41:341
2763045991917,1cyclictest3056-21PK-Backend10:30:220
276304599191,4cyclictest2838606-21apt10:45:000
2763045991911,6cyclictest2847249-21cron11:10:000
276304599190,1cyclictest3056-21PK-Backend10:53:350
276304599190,1cyclictest0-21swapper/012:32:300
276304599190,1cyclictest0-21swapper/010:14:130
276304599190,1cyclictest0-21swapper/008:35:310
276304599190,1cyclictest0-21swapper/008:35:310
276304599190,18cyclictest0-21swapper/012:27:360
276304599190,18cyclictest0-21swapper/008:49:030
2763054991816,1cyclictest3056-21PK-Backend11:39:303
2763054991816,1cyclictest3056-21PK-Backend09:37:563
2763054991816,1cyclictest0-21swapper/309:18:573
2763051991816,1cyclictest0-21swapper/212:34:122
2763051991810,6cyclictest2864110-21awk11:55:162
2763046991816,1cyclictest0-21swapper/108:58:091
2763046991816,1cyclictest0-21swapper/107:58:071
2763046991814,3cyclictest2852722-21idleruntime-cro11:24:591
2763046991813,3cyclictest2799235-21latency08:50:171
276304699181,15cyclictest2852992-21apt11:25:121
2763046991811,5cyclictest2848033-21expr11:10:131
2763046991811,5cyclictest2793456-21date08:35:011
2763046991810,6cyclictest2872860-21cat12:20:151
2763046991810,6cyclictest2860412-21awk11:45:161
2763046991810,6cyclictest2827566-21python310:10:201
276304699180,1cyclictest0-21swapper/107:27:381
276304599189,7cyclictest2852330-21users11:20:220
276304599188,8cyclictest2766335-21needreboot07:15:170
276304599183,13cyclictest2870344-21apt-config12:15:020
2763045991815,2cyclictest0-21swapper/011:25:100
2763045991810,6cyclictest13-21ksoftirqd/010:05:130
276304599180,1cyclictest0-21swapper/008:12:540
2763054991715,1cyclictest3056-21PK-Backend08:48:153
276305499171,3cyclictest2855470-21ntp_states11:30:173
276305499171,3cyclictest2855470-21ntp_states11:30:163
2763054991711,5cyclictest2860848-21wc11:45:223
2763051991711,5cyclictest2825948-21sed10:05:242
276304699179,6cyclictest2856354-21date11:35:011
276304699179,6cyclictest2856354-21date11:35:011
276304699178,7cyclictest2830757-21memory10:20:181
276304699178,7cyclictest1-21systemd10:05:001
276304699172,13cyclictest2793618-21sort08:35:121
276304699172,13cyclictest2793618-21sort08:35:121
2763046991711,5cyclictest2784063-21sed08:05:231
2763046991711,5cyclictest2771887-21sed07:30:221
2763046991711,5cyclictest2771887-21sed07:30:221
2763046991711,4cyclictest2858123-21sed11:40:011
2763046991710,5cyclictest2840937-21cat10:50:121
276304699170,16cyclictest3056-21PK-Backend08:24:511
2763045991715,1cyclictest0-21swapper/009:54:200
2763045991715,1cyclictest0-21swapper/009:41:490
276304599171,12cyclictest2775052-21chrt07:40:170
2763045991710,5cyclictest13-21ksoftirqd/008:15:110
276305499166,2cyclictest42-21ksoftirqd/307:19:183
2763054991610,4cyclictest42-21ksoftirqd/311:40:203
276305499160,3cyclictest2853551-21cat11:25:173
276305499160,3cyclictest141rcu_preempt11:51:003
2763051991612,3cyclictest2849658-21tr11:15:112
276304699169,5cyclictest2772949-21expr07:35:141
2763046991614,1cyclictest0-21swapper/107:54:151
2763046991613,2cyclictest2825765-21proc_pri10:05:221
2763046991612,2cyclictest2823172-21latency_hist10:00:001
2763046991612,2cyclictest2818812-21open_inodes09:45:191
276304699160,15cyclictest26-21ksoftirqd/110:15:281
2763045991610,3cyclictest436-20systemd-journal07:25:000
2763054991512,2cyclictest3056-21PK-Backend11:20:373
2763054991512,2cyclictest3056-21PK-Backend11:19:223
2763051991511,3cyclictest2851468-21cstates11:20:132
2763051991510,4cyclictest978-21snmpd11:07:152
2763046991510,3cyclictest2838598-21latency_hist10:45:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional